2023-04-01 03:56:11,191 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e 2023-04-01 03:56:11,202 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-01 03:56:11,216 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-01 03:56:11,217 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648, deleteOnExit=true 2023-04-01 03:56:11,217 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-01 03:56:11,218 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/test.cache.data in system properties and HBase conf 2023-04-01 03:56:11,218 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/hadoop.tmp.dir in system properties and HBase conf 2023-04-01 03:56:11,219 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/hadoop.log.dir in system properties and HBase conf 2023-04-01 03:56:11,220 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-01 03:56:11,220 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-01 03:56:11,220 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-01 03:56:11,316 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-01 03:56:11,650 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-01 03:56:11,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-01 03:56:11,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-01 03:56:11,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-01 03:56:11,655 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-01 03:56:11,655 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-01 03:56:11,655 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-01 03:56:11,655 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-01 03:56:11,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-01 03:56:11,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-01 03:56:11,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/nfs.dump.dir in system properties and HBase conf 2023-04-01 03:56:11,656 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/java.io.tmpdir in system properties and HBase conf 2023-04-01 03:56:11,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-01 03:56:11,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-01 03:56:11,657 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-01 03:56:12,155 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-01 03:56:12,159 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-01 03:56:12,435 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-01 03:56:12,619 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-01 03:56:12,633 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-01 03:56:12,666 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-01 03:56:12,692 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/java.io.tmpdir/Jetty_localhost_localdomain_35185_hdfs____.1fm862/webapp 2023-04-01 03:56:12,809 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:35185 2023-04-01 03:56:12,842 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-01 03:56:12,842 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-01 03:56:13,154 WARN [Listener at localhost.localdomain/32871] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-01 03:56:13,236 WARN [Listener at localhost.localdomain/32871] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-01 03:56:13,254 WARN [Listener at localhost.localdomain/32871] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-01 03:56:13,260 INFO [Listener at localhost.localdomain/32871] log.Slf4jLog(67): jetty-6.1.26 2023-04-01 03:56:13,265 INFO [Listener at localhost.localdomain/32871] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/java.io.tmpdir/Jetty_localhost_35355_datanode____savw0o/webapp 2023-04-01 03:56:13,342 INFO [Listener at localhost.localdomain/32871] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35355 2023-04-01 03:56:13,604 WARN [Listener at localhost.localdomain/39075] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-01 03:56:13,955 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa9a599200a73eebd: Processing first storage report for DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de from datanode 1765434b-9800-4ee8-9fa5-09d8b0e4da2f 2023-04-01 03:56:13,956 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa9a599200a73eebd: from storage DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de node DatanodeRegistration(127.0.0.1:38933, datanodeUuid=1765434b-9800-4ee8-9fa5-09d8b0e4da2f, infoPort=36203, infoSecurePort=0, ipcPort=39075, storageInfo=lv=-57;cid=testClusterID;nsid=726634220;c=1680321372222), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-01 03:56:13,956 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa9a599200a73eebd: Processing first storage report for DS-1861b264-1556-48d3-a38d-ff07e13d49e9 from datanode 1765434b-9800-4ee8-9fa5-09d8b0e4da2f 2023-04-01 03:56:13,957 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa9a599200a73eebd: from storage DS-1861b264-1556-48d3-a38d-ff07e13d49e9 node DatanodeRegistration(127.0.0.1:38933, datanodeUuid=1765434b-9800-4ee8-9fa5-09d8b0e4da2f, infoPort=36203, infoSecurePort=0, ipcPort=39075, storageInfo=lv=-57;cid=testClusterID;nsid=726634220;c=1680321372222), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-01 03:56:14,116 DEBUG [Listener at localhost.localdomain/39075] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e 2023-04-01 03:56:14,163 INFO [Listener at localhost.localdomain/39075] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648/zookeeper_0, clientPort=50457, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-01 03:56:14,173 INFO [Listener at localhost.localdomain/39075] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50457 2023-04-01 03:56:14,181 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:14,183 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:14,794 INFO [Listener at localhost.localdomain/39075] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a with version=8 2023-04-01 03:56:14,795 INFO [Listener at localhost.localdomain/39075] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/hbase-staging 2023-04-01 03:56:15,032 INFO [Listener at localhost.localdomain/39075] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-01 03:56:15,379 INFO [Listener at localhost.localdomain/39075] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-01 03:56:15,403 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,404 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,404 INFO [Listener at localhost.localdomain/39075] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-01 03:56:15,404 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,404 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-01 03:56:15,515 INFO [Listener at localhost.localdomain/39075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-01 03:56:15,572 DEBUG [Listener at localhost.localdomain/39075] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-01 03:56:15,647 INFO [Listener at localhost.localdomain/39075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34393 2023-04-01 03:56:15,655 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:15,657 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:15,674 INFO [Listener at localhost.localdomain/39075] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34393 connecting to ZooKeeper ensemble=127.0.0.1:50457 2023-04-01 03:56:15,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:343930x0, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-01 03:56:15,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34393-0x1006aab7ff40000 connected 2023-04-01 03:56:15,740 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 03:56:15,741 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 03:56:15,744 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-01 03:56:15,750 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34393 2023-04-01 03:56:15,751 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34393 2023-04-01 03:56:15,751 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34393 2023-04-01 03:56:15,751 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34393 2023-04-01 03:56:15,752 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34393 2023-04-01 03:56:15,757 INFO [Listener at localhost.localdomain/39075] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a, hbase.cluster.distributed=false 2023-04-01 03:56:15,817 INFO [Listener at localhost.localdomain/39075] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-01 03:56:15,817 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,817 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,817 INFO [Listener at localhost.localdomain/39075] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-01 03:56:15,817 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 03:56:15,818 INFO [Listener at localhost.localdomain/39075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-01 03:56:15,821 INFO [Listener at localhost.localdomain/39075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-01 03:56:15,824 INFO [Listener at localhost.localdomain/39075] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36251 2023-04-01 03:56:15,826 INFO [Listener at localhost.localdomain/39075] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-01 03:56:15,831 DEBUG [Listener at localhost.localdomain/39075] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-01 03:56:15,832 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:15,834 INFO [Listener at localhost.localdomain/39075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:15,836 INFO [Listener at localhost.localdomain/39075] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36251 connecting to ZooKeeper ensemble=127.0.0.1:50457 2023-04-01 03:56:15,846 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:362510x0, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-01 03:56:15,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36251-0x1006aab7ff40001 connected 2023-04-01 03:56:15,848 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 03:56:15,849 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 03:56:15,850 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-01 03:56:15,850 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36251 2023-04-01 03:56:15,850 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36251 2023-04-01 03:56:15,851 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36251 2023-04-01 03:56:15,851 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36251 2023-04-01 03:56:15,851 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36251 2023-04-01 03:56:15,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:15,864 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-01 03:56:15,865 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:15,884 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-01 03:56:15,884 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-01 03:56:15,884 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:15,885 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-01 03:56:15,886 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,34393,1680321374904 from backup master directory 2023-04-01 03:56:15,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-01 03:56:15,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:15,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-01 03:56:15,891 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-01 03:56:15,891 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:15,894 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-01 03:56:15,895 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-01 03:56:15,973 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/hbase.id with ID: 99f6de51-a43d-4825-b24b-127b7e1478e6 2023-04-01 03:56:16,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 03:56:16,029 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:16,063 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0f9fda4d to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:16,092 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@617c3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:16,111 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:16,113 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-01 03:56:16,127 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-01 03:56:16,127 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-01 03:56:16,130 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:16,133 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:16,134 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 03:56:16,160 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store-tmp 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-01 03:56:16,592 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:56:16,592 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:56:16,592 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 03:56:16,594 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/WALs/jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:16,611 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34393%2C1680321374904, suffix=, logDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/WALs/jenkins-hbase20.apache.org,34393,1680321374904, archiveDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/oldWALs, maxLogs=10 2023-04-01 03:56:16,658 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK] 2023-04-01 03:56:16,665 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:16,715 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/WALs/jenkins-hbase20.apache.org,34393,1680321374904/jenkins-hbase20.apache.org%2C34393%2C1680321374904.1680321376619 2023-04-01 03:56:16,716 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK]] 2023-04-01 03:56:16,716 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:16,717 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:16,719 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,720 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,770 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,776 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-01 03:56:16,796 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-01 03:56:16,806 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:16,811 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,813 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,829 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 03:56:16,834 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:56:16,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65479781, jitterRate=-0.02427522838115692}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-01 03:56:16,835 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 03:56:16,836 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-01 03:56:16,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-01 03:56:16,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-01 03:56:16,857 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-01 03:56:16,858 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-01 03:56:16,884 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-04-01 03:56:16,884 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-01 03:56:16,907 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-01 03:56:16,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-01 03:56:16,936 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-01 03:56:16,939 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-01 03:56:16,942 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-01 03:56:16,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-01 03:56:16,951 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-01 03:56:16,954 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:16,955 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-01 03:56:16,956 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-01 03:56:16,968 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-01 03:56:16,972 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-01 03:56:16,972 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-01 03:56:16,972 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:16,972 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,34393,1680321374904, sessionid=0x1006aab7ff40000, setting cluster-up flag (Was=false) 2023-04-01 03:56:16,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:16,988 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-01 03:56:16,989 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:16,994 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:16,998 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-01 03:56:16,999 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:17,001 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.hbase-snapshot/.tmp 2023-04-01 03:56:17,056 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(952): ClusterId : 99f6de51-a43d-4825-b24b-127b7e1478e6 2023-04-01 03:56:17,060 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-01 03:56:17,083 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-01 03:56:17,083 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-01 03:56:17,095 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-01 03:56:17,100 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-01 03:56:17,101 DEBUG [RS:0;jenkins-hbase20:36251] zookeeper.ReadOnlyZKClient(139): Connect 0x76311fa0 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:17,106 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-01 03:56:17,106 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-01 03:56:17,106 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-01 03:56:17,107 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-01 03:56:17,107 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-01 03:56:17,107 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,107 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-01 03:56:17,107 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,108 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680321407108 2023-04-01 03:56:17,110 DEBUG [RS:0;jenkins-hbase20:36251] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78f6ff16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:17,111 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-01 03:56:17,111 DEBUG [RS:0;jenkins-hbase20:36251] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7cc974d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-01 03:56:17,114 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-01 03:56:17,114 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-01 03:56:17,120 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:17,123 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-01 03:56:17,131 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-01 03:56:17,131 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-01 03:56:17,132 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-01 03:56:17,132 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-01 03:56:17,133 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,133 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:36251 2023-04-01 03:56:17,134 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-01 03:56:17,135 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-01 03:56:17,136 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-01 03:56:17,137 INFO [RS:0;jenkins-hbase20:36251] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-01 03:56:17,137 INFO [RS:0;jenkins-hbase20:36251] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-01 03:56:17,137 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1023): About to register with Master. 2023-04-01 03:56:17,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-01 03:56:17,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-01 03:56:17,140 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,34393,1680321374904 with isa=jenkins-hbase20.apache.org/148.251.75.209:36251, startcode=1680321375816 2023-04-01 03:56:17,143 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680321377142,5,FailOnTimeoutGroup] 2023-04-01 03:56:17,143 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680321377143,5,FailOnTimeoutGroup] 2023-04-01 03:56:17,143 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,144 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-01 03:56:17,146 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,146 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,149 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:17,149 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:17,150 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a 2023-04-01 03:56:17,159 DEBUG [RS:0;jenkins-hbase20:36251] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-01 03:56:17,199 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50377, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-01 03:56:17,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,222 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a 2023-04-01 03:56:17,222 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:32871 2023-04-01 03:56:17,223 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-01 03:56:17,227 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 03:56:17,228 DEBUG [RS:0;jenkins-hbase20:36251] zookeeper.ZKUtil(162): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,229 WARN [RS:0;jenkins-hbase20:36251] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-01 03:56:17,229 INFO [RS:0;jenkins-hbase20:36251] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 03:56:17,230 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/WALs/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,230 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36251,1680321375816] 2023-04-01 03:56:17,240 DEBUG [RS:0;jenkins-hbase20:36251] zookeeper.ZKUtil(162): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,249 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-01 03:56:17,256 INFO [RS:0;jenkins-hbase20:36251] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-01 03:56:17,274 INFO [RS:0;jenkins-hbase20:36251] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-01 03:56:17,277 INFO [RS:0;jenkins-hbase20:36251] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-01 03:56:17,277 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,278 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-01 03:56:17,284 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,285 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,285 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,285 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,286 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,287 DEBUG [RS:0;jenkins-hbase20:36251] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-01 03:56:17,288 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,288 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,288 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,301 INFO [RS:0;jenkins-hbase20:36251] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-01 03:56:17,303 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36251,1680321375816-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:17,316 INFO [RS:0;jenkins-hbase20:36251] regionserver.Replication(203): jenkins-hbase20.apache.org,36251,1680321375816 started 2023-04-01 03:56:17,316 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,36251,1680321375816, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36251, sessionid=0x1006aab7ff40001 2023-04-01 03:56:17,317 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-01 03:56:17,317 DEBUG [RS:0;jenkins-hbase20:36251] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,317 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36251,1680321375816' 2023-04-01 03:56:17,317 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:17,318 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:17,318 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-01 03:56:17,318 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-01 03:56:17,318 DEBUG [RS:0;jenkins-hbase20:36251] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:17,319 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36251,1680321375816' 2023-04-01 03:56:17,319 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-01 03:56:17,319 DEBUG [RS:0;jenkins-hbase20:36251] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-01 03:56:17,320 DEBUG [RS:0;jenkins-hbase20:36251] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-01 03:56:17,320 INFO [RS:0;jenkins-hbase20:36251] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-01 03:56:17,320 INFO [RS:0;jenkins-hbase20:36251] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-01 03:56:17,439 INFO [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36251%2C1680321375816, suffix=, logDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/WALs/jenkins-hbase20.apache.org,36251,1680321375816, archiveDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/oldWALs, maxLogs=32 2023-04-01 03:56:17,455 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK] 2023-04-01 03:56:17,458 INFO [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/WALs/jenkins-hbase20.apache.org,36251,1680321375816/jenkins-hbase20.apache.org%2C36251%2C1680321375816.1680321377441 2023-04-01 03:56:17,458 DEBUG [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK]] 2023-04-01 03:56:17,562 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:17,565 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-01 03:56:17,568 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/info 2023-04-01 03:56:17,569 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-01 03:56:17,571 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:17,571 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-01 03:56:17,575 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 03:56:17,576 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-01 03:56:17,578 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:17,578 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-01 03:56:17,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/table 2023-04-01 03:56:17,582 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-01 03:56:17,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:17,585 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740 2023-04-01 03:56:17,586 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740 2023-04-01 03:56:17,590 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:56:17,592 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-01 03:56:17,596 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:56:17,597 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75230542, jitterRate=0.12102243304252625}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:56:17,598 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-01 03:56:17,598 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-01 03:56:17,598 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-01 03:56:17,598 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-01 03:56:17,598 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-01 03:56:17,598 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-01 03:56:17,600 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-01 03:56:17,600 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-01 03:56:17,607 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-01 03:56:17,607 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-01 03:56:17,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-01 03:56:17,628 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-01 03:56:17,630 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-01 03:56:17,782 DEBUG [jenkins-hbase20:34393] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-01 03:56:17,785 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36251,1680321375816, state=OPENING 2023-04-01 03:56:17,792 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-01 03:56:17,794 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:17,794 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-01 03:56:17,797 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:17,985 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:18,005 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-01 03:56:18,010 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-01 03:56:18,022 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-01 03:56:18,022 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 03:56:18,026 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36251%2C1680321375816.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/WALs/jenkins-hbase20.apache.org,36251,1680321375816, archiveDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/oldWALs, maxLogs=32 2023-04-01 03:56:18,043 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK] 2023-04-01 03:56:18,047 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/WALs/jenkins-hbase20.apache.org,36251,1680321375816/jenkins-hbase20.apache.org%2C36251%2C1680321375816.meta.1680321378028.meta 2023-04-01 03:56:18,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38933,DS-f00fdcbd-94d5-4ab9-b0ef-33bed924d3de,DISK]] 2023-04-01 03:56:18,048 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:18,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-01 03:56:18,065 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-01 03:56:18,069 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-01 03:56:18,074 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-01 03:56:18,075 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:18,075 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-01 03:56:18,075 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-01 03:56:18,078 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-01 03:56:18,080 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/info 2023-04-01 03:56:18,080 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/info 2023-04-01 03:56:18,081 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-01 03:56:18,082 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:18,082 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-01 03:56:18,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 03:56:18,083 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 03:56:18,084 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-01 03:56:18,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:18,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-01 03:56:18,086 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/table 2023-04-01 03:56:18,086 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/table 2023-04-01 03:56:18,087 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-01 03:56:18,088 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:18,090 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740 2023-04-01 03:56:18,092 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740 2023-04-01 03:56:18,096 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:56:18,098 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-01 03:56:18,099 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68152895, jitterRate=0.015557274222373962}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:56:18,099 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-01 03:56:18,110 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680321377976 2023-04-01 03:56:18,128 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-01 03:56:18,129 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-01 03:56:18,129 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36251,1680321375816, state=OPEN 2023-04-01 03:56:18,132 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-01 03:56:18,132 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-01 03:56:18,137 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-01 03:56:18,137 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36251,1680321375816 in 335 msec 2023-04-01 03:56:18,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-01 03:56:18,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 524 msec 2023-04-01 03:56:18,148 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1070 sec 2023-04-01 03:56:18,148 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680321378148, completionTime=-1 2023-04-01 03:56:18,148 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-01 03:56:18,149 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-01 03:56:18,200 DEBUG [hconnection-0x7a6fa178-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:18,204 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:18,225 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-01 03:56:18,225 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680321438225 2023-04-01 03:56:18,225 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680321498225 2023-04-01 03:56:18,225 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 76 msec 2023-04-01 03:56:18,244 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34393,1680321374904-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:18,244 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34393,1680321374904-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:18,245 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34393,1680321374904-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:18,246 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:34393, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:18,246 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-01 03:56:18,253 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-01 03:56:18,259 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-01 03:56:18,260 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:18,269 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-01 03:56:18,272 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:56:18,274 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:56:18,293 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:18,295 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa empty. 2023-04-01 03:56:18,296 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:18,297 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-01 03:56:18,722 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:18,725 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9831deae690ed99cd6a015ff582234aa, NAME => 'hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:56:19,139 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:19,139 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 9831deae690ed99cd6a015ff582234aa, disabling compactions & flushes 2023-04-01 03:56:19,140 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,140 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,140 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. after waiting 0 ms 2023-04-01 03:56:19,140 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,140 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,140 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 9831deae690ed99cd6a015ff582234aa: 2023-04-01 03:56:19,145 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:56:19,161 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680321379148"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321379148"}]},"ts":"1680321379148"} 2023-04-01 03:56:19,185 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:56:19,187 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:56:19,191 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321379187"}]},"ts":"1680321379187"} 2023-04-01 03:56:19,196 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-01 03:56:19,203 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9831deae690ed99cd6a015ff582234aa, ASSIGN}] 2023-04-01 03:56:19,206 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9831deae690ed99cd6a015ff582234aa, ASSIGN 2023-04-01 03:56:19,208 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=9831deae690ed99cd6a015ff582234aa, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:56:19,359 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9831deae690ed99cd6a015ff582234aa, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:19,359 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680321379359"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321379359"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321379359"}]},"ts":"1680321379359"} 2023-04-01 03:56:19,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 9831deae690ed99cd6a015ff582234aa, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:19,529 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,534 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9831deae690ed99cd6a015ff582234aa, NAME => 'hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:19,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:19,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,541 INFO [StoreOpener-9831deae690ed99cd6a015ff582234aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,544 DEBUG [StoreOpener-9831deae690ed99cd6a015ff582234aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/info 2023-04-01 03:56:19,544 DEBUG [StoreOpener-9831deae690ed99cd6a015ff582234aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/info 2023-04-01 03:56:19,545 INFO [StoreOpener-9831deae690ed99cd6a015ff582234aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9831deae690ed99cd6a015ff582234aa columnFamilyName info 2023-04-01 03:56:19,546 INFO [StoreOpener-9831deae690ed99cd6a015ff582234aa-1] regionserver.HStore(310): Store=9831deae690ed99cd6a015ff582234aa/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:19,548 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,556 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:56:19,563 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:56:19,564 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 9831deae690ed99cd6a015ff582234aa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68135200, jitterRate=0.015293598175048828}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-01 03:56:19,564 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 9831deae690ed99cd6a015ff582234aa: 2023-04-01 03:56:19,566 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa., pid=6, masterSystemTime=1680321379520 2023-04-01 03:56:19,572 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,572 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:56:19,574 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9831deae690ed99cd6a015ff582234aa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:19,575 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680321379573"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321379573"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321379573"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321379573"}]},"ts":"1680321379573"} 2023-04-01 03:56:19,587 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-01 03:56:19,587 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 9831deae690ed99cd6a015ff582234aa, server=jenkins-hbase20.apache.org,36251,1680321375816 in 214 msec 2023-04-01 03:56:19,592 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-01 03:56:19,593 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=9831deae690ed99cd6a015ff582234aa, ASSIGN in 384 msec 2023-04-01 03:56:19,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:56:19,595 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321379595"}]},"ts":"1680321379595"} 2023-04-01 03:56:19,599 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-01 03:56:19,602 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:56:19,607 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3420 sec 2023-04-01 03:56:19,675 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-01 03:56:19,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-01 03:56:19,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:56:19,722 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-01 03:56:19,757 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-01 03:56:19,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 56 msec 2023-04-01 03:56:19,787 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-01 03:56:19,802 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-01 03:56:19,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-04-01 03:56:19,825 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-01 03:56:19,828 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-01 03:56:19,829 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.938sec 2023-04-01 03:56:19,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-01 03:56:19,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-01 03:56:19,833 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-01 03:56:19,834 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34393,1680321374904-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-01 03:56:19,834 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34393,1680321374904-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-01 03:56:19,844 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-01 03:56:19,866 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x37e2243f to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:19,871 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@277cdafb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:19,884 DEBUG [hconnection-0x659ed469-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:19,895 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:19,905 INFO [Listener at localhost.localdomain/39075] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:56:19,930 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=204, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=197, ProcessCount=167, AvailableMemoryMB=8461 2023-04-01 03:56:19,945 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:56:19,949 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:56:19,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:56:19,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:19,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:19,965 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:56:19,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:56:19,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-01 03:56:19,972 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:19,974 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 empty. 2023-04-01 03:56:19,975 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:19,976 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:56:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 03:56:19,995 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:19,997 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d055bdf2a6c005123440ac61e55072e8, NAME => 'TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:56:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 03:56:20,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 03:56:20,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:20,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d055bdf2a6c005123440ac61e55072e8, disabling compactions & flushes 2023-04-01 03:56:20,410 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,410 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. after waiting 0 ms 2023-04-01 03:56:20,411 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,411 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,411 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:20,415 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:56:20,417 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321380417"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321380417"}]},"ts":"1680321380417"} 2023-04-01 03:56:20,419 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:56:20,421 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:56:20,421 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321380421"}]},"ts":"1680321380421"} 2023-04-01 03:56:20,423 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:56:20,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, ASSIGN}] 2023-04-01 03:56:20,431 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, ASSIGN 2023-04-01 03:56:20,432 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:56:20,584 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d055bdf2a6c005123440ac61e55072e8, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:20,584 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321380584"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321380584"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321380584"}]},"ts":"1680321380584"} 2023-04-01 03:56:20,588 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:20,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 03:56:20,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d055bdf2a6c005123440ac61e55072e8, NAME => 'TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:20,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:20,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,760 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,763 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A 2023-04-01 03:56:20,763 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A 2023-04-01 03:56:20,764 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:20,765 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d055bdf2a6c005123440ac61e55072e8 columnFamilyName A 2023-04-01 03:56:20,766 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(310): Store=d055bdf2a6c005123440ac61e55072e8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:20,766 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,768 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B 2023-04-01 03:56:20,768 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B 2023-04-01 03:56:20,769 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:20,769 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d055bdf2a6c005123440ac61e55072e8 columnFamilyName B 2023-04-01 03:56:20,770 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(310): Store=d055bdf2a6c005123440ac61e55072e8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:20,770 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,772 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C 2023-04-01 03:56:20,772 DEBUG [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C 2023-04-01 03:56:20,773 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:20,773 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d055bdf2a6c005123440ac61e55072e8 columnFamilyName C 2023-04-01 03:56:20,774 INFO [StoreOpener-d055bdf2a6c005123440ac61e55072e8-1] regionserver.HStore(310): Store=d055bdf2a6c005123440ac61e55072e8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:20,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:56:20,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:20,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:56:20,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened d055bdf2a6c005123440ac61e55072e8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60563089, jitterRate=-0.09753964841365814}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:56:20,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:20,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., pid=11, masterSystemTime=1680321380743 2023-04-01 03:56:20,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:20,798 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d055bdf2a6c005123440ac61e55072e8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:20,798 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321380798"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321380798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321380798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321380798"}]},"ts":"1680321380798"} 2023-04-01 03:56:20,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-01 03:56:20,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 in 213 msec 2023-04-01 03:56:20,807 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-01 03:56:20,807 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, ASSIGN in 376 msec 2023-04-01 03:56:20,809 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:56:20,809 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321380809"}]},"ts":"1680321380809"} 2023-04-01 03:56:20,811 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:56:20,814 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:56:20,816 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 854 msec 2023-04-01 03:56:21,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 03:56:21,097 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-01 03:56:21,101 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x7a33cf3a to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,106 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d9211a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,109 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,113 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,116 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:56:21,120 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:56:21,130 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6931edc7 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,164 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7318d323, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,164 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x0faf312d to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,183 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c22ed00, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,184 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x4f7664e5 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,191 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13eac73b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,192 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x51210e37 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,197 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@689ad03f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,197 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x32ee6250 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,203 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f62ec6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,204 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x40d2a406 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,208 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b186e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,209 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x26c4dd91 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,231 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7363bfdb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,232 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3a15fbcc to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,260 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54e60632, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,260 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x23c201e2 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:21,268 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@369c55cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:21,276 DEBUG [hconnection-0x217990b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,276 DEBUG [hconnection-0x4f512920-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,276 DEBUG [hconnection-0x6d6b30dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,276 DEBUG [hconnection-0x5b9b7798-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,277 DEBUG [hconnection-0x33f7e9c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,285 DEBUG [hconnection-0x6ff5cb68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,286 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,286 DEBUG [hconnection-0x1440f91b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,286 DEBUG [hconnection-0x7328ed9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,290 DEBUG [hconnection-0xac8a7e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:21,291 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,295 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:21,297 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,299 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,303 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43566, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,308 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,310 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,316 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:21,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:21,332 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:21,332 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:21,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:21,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:21,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:21,337 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:21,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:21,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:21,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:21,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:21,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:21,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:21,349 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:21,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:21,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:21,371 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:21,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:21,373 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 03:56:21,392 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:21,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:21,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:21,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:21,395 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:21,395 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:21,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:21,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321381500, max timestamp=1680321381522Num uniques 859; 2023-04-01 03:56:21,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:56:21,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321381500, max timestamp=1680321381518Num uniques 859; 2023-04-01 03:56:21,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:56:21,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321381500, max timestamp=1680321381528Num uniques 859; 2023-04-01 03:56:21,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:56:21,603 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e0274ed747ae468989a2be5410259b51 2023-04-01 03:56:21,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321441627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321381618, max timestamp=1680321381630Num uniques 1018; 2023-04-01 03:56:21,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-01 03:56:21,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321381618, max timestamp=1680321381630Num uniques 1018; 2023-04-01 03:56:21,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-01 03:56:21,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321381618, max timestamp=1680321381627Num uniques 1009; 2023-04-01 03:56:21,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 03:56:21,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680321381680, max timestamp=1680321381691Num uniques 1027; 2023-04-01 03:56:21,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-01 03:56:21,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680321381680, max timestamp=1680321381691Num uniques 1009; 2023-04-01 03:56:21,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-01 03:56:21,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321381680, max timestamp=1680321381691Num uniques 1009; 2023-04-01 03:56:21,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 03:56:21,762 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/383944d33c4c4ed68bd77fac16231e35 2023-04-01 03:56:21,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8507, totalHeapSize=19584, min timestamp=1680321381734, max timestamp=1680321381772Num uniques 1009; 2023-04-01 03:56:21,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46205, new segment data size=8507, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-01 03:56:21,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680321381734, max timestamp=1680321381770Num uniques 1036; 2023-04-01 03:56:21,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-01 03:56:21,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680321381734, max timestamp=1680321381768Num uniques 1036; 2023-04-01 03:56:21,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-01 03:56:21,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321441815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:21,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:21,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321441819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321441819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680321381809, max timestamp=1680321381819Num uniques 1045; 2023-04-01 03:56:21,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-01 03:56:21,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680321381809, max timestamp=1680321381819Num uniques 1045; 2023-04-01 03:56:21,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680321381809, max timestamp=1680321381818Num uniques 1045; 2023-04-01 03:56:21,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-01 03:56:21,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-01 03:56:21,842 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc 2023-04-01 03:56:21,857 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e0274ed747ae468989a2be5410259b51 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51 2023-04-01 03:56:21,872 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51, entries=150, sequenceid=10, filesize=11.6 K 2023-04-01 03:56:21,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/383944d33c4c4ed68bd77fac16231e35 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35 2023-04-01 03:56:21,911 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35, entries=150, sequenceid=10, filesize=11.6 K 2023-04-01 03:56:21,918 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc 2023-04-01 03:56:21,931 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc, entries=150, sequenceid=10, filesize=11.6 K 2023-04-01 03:56:21,938 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=80.51 KB/82440 for d055bdf2a6c005123440ac61e55072e8 in 564ms, sequenceid=10, compaction requested=false 2023-04-01 03:56:21,940 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-01 03:56:21,945 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:21,945 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:21,945 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:21,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:21,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,947 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:21,947 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:21,949 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,949 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:21,951 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,951 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:21,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=87.22 KB heapSize=242.16 KB 2023-04-01 03:56:21,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:21,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:21,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:21,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:21,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:21,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ee490a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:21,960 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:21,960 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,972 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:21,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:21,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:21,979 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:21,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:21,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:21,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:21,979 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:21,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:21,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,037 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:22,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:22,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:22,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:22,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/965ddf2678d34eb0897ddc1440e68895 2023-04-01 03:56:22,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:22,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321442095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321442095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321442103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321442117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:22,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321442145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:22,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/71d0730e2e0b41459fc2bd05779ab9ac 2023-04-01 03:56:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:22,242 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:22,242 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:22,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:22,256 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,256 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321442248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321442249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321442250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321442248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:22,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:22,294 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:22,294 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:22,299 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,299 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:22,299 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:22,299 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321442291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:22,311 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,312 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,312 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a263343[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:22,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:22,318 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,320 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:22,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:22,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:22,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:22,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:22,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:22,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:22,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:22,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:22,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,379 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:22,380 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,383 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:22,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321442462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321442463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321442464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321442466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:22,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:22,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321442512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:22,516 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:22,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:22,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:22,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:22,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:22,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:22,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:22,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:22,536 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,536 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:22,536 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:22,536 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:22,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:22,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:22,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:22,555 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61089509[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:22,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:22,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:22,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:22,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:22,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:22,571 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:22,577 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:22,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:22,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:22,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:22,578 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:22,583 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,583 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,583 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,585 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:22,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:22,636 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,636 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:22,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/71ee50bf32c3483bbd756d31572c7b21 2023-04-01 03:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/965ddf2678d34eb0897ddc1440e68895 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895 2023-04-01 03:56:22,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895, entries=200, sequenceid=108, filesize=13.9 K 2023-04-01 03:56:22,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:22,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:22,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/71d0730e2e0b41459fc2bd05779ab9ac as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac 2023-04-01 03:56:22,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:22,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac, entries=150, sequenceid=108, filesize=11.6 K 2023-04-01 03:56:22,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/71ee50bf32c3483bbd756d31572c7b21 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21 2023-04-01 03:56:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:22,791 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:22,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:22,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:22,794 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,794 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,796 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321442767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321442772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21, entries=150, sequenceid=108, filesize=11.6 K 2023-04-01 03:56:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321442770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~242.11 KB/247920, currentSize=93.93 KB/96180 for d055bdf2a6c005123440ac61e55072e8 in 853ms, sequenceid=108, compaction requested=false 2023-04-01 03:56:22,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321442772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:22,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:22,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:22,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:22,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:56:22,822 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:22,822 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:22,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:22,822 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:22,823 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:22,823 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:22,823 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:22,823 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:22,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:22,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:22,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:22,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:22,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:22,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,829 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516ad14b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:22,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:22,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,836 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,836 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:22,836 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:22,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:22,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:22,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:22,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:22,848 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:22,865 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:22,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:22,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:22,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:22,866 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:22,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:22,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:22,874 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:22,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:22,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:22,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:22,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:22,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:22,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:22,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:22,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:22,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:22,908 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:22,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:22,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:22,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321442913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:23,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:23,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321443022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:23,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:23,050 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:23,050 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:23,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:23,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:23,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:23,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:23,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:23,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:23,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:23,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:23,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:23,057 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:23,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,067 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,068 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,068 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:23,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cc6475[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:23,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:23,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,075 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,076 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:23,076 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:23,076 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,076 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:23,076 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:23,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:23,078 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:23,101 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:23,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:23,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:23,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:23,101 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:23,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,104 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:23,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:23,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,120 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:23,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:23,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,223 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:23,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:23,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321443226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:23,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:23,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:23,238 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:23,238 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:23,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:23,244 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,244 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,245 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,245 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:23,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:23,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:23,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:23,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:23,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:23,267 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,267 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:23,267 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:23,267 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:23,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:23,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:23,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:23,279 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a5bf924[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,284 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:23,284 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,284 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:23,287 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:23,291 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:23,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:23,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:23,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:23,291 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:23,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,294 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321443308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321443310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321443312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:23,314 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:23,314 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321443317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/c7a40749a83140d89e40d2b32d2d820d 2023-04-01 03:56:23,414 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:23,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:23,430 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:23,430 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:23,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:23,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:23,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:23,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:23,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:23,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,466 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:23,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:23,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:23,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:23,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:23,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:23,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5e8bbb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:23,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:23,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:23,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:23,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,502 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,502 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:23,509 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:23,509 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:23,509 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:23,509 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:23,509 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:23,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,515 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,528 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,529 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:23,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:23,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:23,529 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:23,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:23,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:23,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321443532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,602 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:56:23,629 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:23,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:23,658 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:23,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:23,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:23,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,661 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:23,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:23,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:23,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,676 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:23,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:23,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:23,680 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,680 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:23,680 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:23,680 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:23,692 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,693 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,693 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:23,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b9f992[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:23,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:23,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,717 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:23,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:23,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:23,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:23,719 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:23,719 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,719 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:23,719 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:23,741 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:23,741 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:23,741 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:23,741 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:23,741 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:23,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,752 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,773 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-01 03:56:23,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:23,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:23,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:23,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:23,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,778 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-01 03:56:23,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:23,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:23,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:23,889 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:23,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,892 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:23,892 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:23,893 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,893 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/6449d9639b8a418e8037bd82c5004661 2023-04-01 03:56:23,895 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,898 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:23,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:23,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:23,905 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:23,905 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:23,906 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:23,906 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:23,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:23,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:23,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:23,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:23,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,909 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a64c13c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:23,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:23,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,918 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,919 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:23,919 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:23,919 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:23,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:23,921 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:23,933 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:23,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:23,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:23,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:23,933 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:23,936 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:23,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:23,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:23,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:23,950 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:23,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:23,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:23,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:23,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:23,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:23,955 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:23,956 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:23,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:23,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/037506c5ee914341a6fad2057d8af12f 2023-04-01 03:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/c7a40749a83140d89e40d2b32d2d820d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d 2023-04-01 03:56:24,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d, entries=200, sequenceid=126, filesize=13.9 K 2023-04-01 03:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/6449d9639b8a418e8037bd82c5004661 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661 2023-04-01 03:56:24,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661, entries=150, sequenceid=126, filesize=11.6 K 2023-04-01 03:56:24,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/037506c5ee914341a6fad2057d8af12f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f 2023-04-01 03:56:24,056 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:24,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:24,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321444054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:24,066 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:24,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,067 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:24,067 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:24,067 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f, entries=150, sequenceid=126, filesize=11.6 K 2023-04-01 03:56:24,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:24,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:24,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d055bdf2a6c005123440ac61e55072e8 in 1248ms, sequenceid=126, compaction requested=true 2023-04-01 03:56:24,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:24,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:24,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:24,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,083 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:24,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:24,084 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:24,084 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:24,084 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:24,084 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:24,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:24,084 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,086 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:24,086 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:56:24,086 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:24,087 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,087 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:24,088 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,088 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:24,088 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,094 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:24,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:24,096 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:24,096 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,096 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=34.8 K 2023-04-01 03:56:24,097 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:24,098 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,098 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=39.4 K 2023-04-01 03:56:24,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e0274ed747ae468989a2be5410259b51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321381324 2023-04-01 03:56:24,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 383944d33c4c4ed68bd77fac16231e35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321381324 2023-04-01 03:56:24,104 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 965ddf2678d34eb0897ddc1440e68895, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680321381845 2023-04-01 03:56:24,105 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 71d0730e2e0b41459fc2bd05779ab9ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680321381871 2023-04-01 03:56:24,106 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6449d9639b8a418e8037bd82c5004661, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:24,109 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c7a40749a83140d89e40d2b32d2d820d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:24,176 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/febdb947f7e140b0bedb0a9aaa842179 2023-04-01 03:56:24,178 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#10 average throughput is 0.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:24,183 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#11 average throughput is 0.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:24,246 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/c5677728ea984a73ab6af7136172897e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c5677728ea984a73ab6af7136172897e 2023-04-01 03:56:24,259 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/31c3aca2c49242e3a11f4bd7c2d38ce8 2023-04-01 03:56:24,265 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/964f60818e044232b7c1bf4b00d06bb2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/964f60818e044232b7c1bf4b00d06bb2 2023-04-01 03:56:24,298 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into c5677728ea984a73ab6af7136172897e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:24,299 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:24,299 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 964f60818e044232b7c1bf4b00d06bb2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:24,299 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321384083; duration=0sec 2023-04-01 03:56:24,305 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:24,305 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:24,307 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:24,308 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:24,308 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,308 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=34.8 K 2023-04-01 03:56:24,308 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1a42a4f61cec4ab6a6d6aa02ac6e13dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321381324 2023-04-01 03:56:24,309 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 71ee50bf32c3483bbd756d31572c7b21, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680321381871 2023-04-01 03:56:24,309 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 037506c5ee914341a6fad2057d8af12f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:24,311 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:24,322 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321384072; duration=0sec 2023-04-01 03:56:24,322 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:24,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:24,334 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/13dc9e7bc380458ba83b43d3c5a8e215 2023-04-01 03:56:24,342 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#14 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:24,346 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/febdb947f7e140b0bedb0a9aaa842179 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179 2023-04-01 03:56:24,364 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179, entries=150, sequenceid=140, filesize=11.8 K 2023-04-01 03:56:24,367 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/31c3aca2c49242e3a11f4bd7c2d38ce8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8 2023-04-01 03:56:24,381 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8, entries=150, sequenceid=140, filesize=11.8 K 2023-04-01 03:56:24,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/13dc9e7bc380458ba83b43d3c5a8e215 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215 2023-04-01 03:56:24,401 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/e975b45d61354b17ac39ae3c93618e9e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e975b45d61354b17ac39ae3c93618e9e 2023-04-01 03:56:24,407 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215, entries=150, sequenceid=140, filesize=11.8 K 2023-04-01 03:56:24,416 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=46.96 KB/48090 for d055bdf2a6c005123440ac61e55072e8 in 330ms, sequenceid=140, compaction requested=false 2023-04-01 03:56:24,416 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:24,416 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:24,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:56:24,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:24,418 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,418 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:24,418 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,418 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:24,418 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:24,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:24,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:24,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:24,421 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,425 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,425 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,442 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into e975b45d61354b17ac39ae3c93618e9e(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,444 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:24,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,445 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321384086; duration=0sec 2023-04-01 03:56:24,445 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f53978d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:24,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:24,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:24,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:24,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:24,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:24,450 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:24,453 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:24,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:24,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:24,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:24,453 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:24,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,477 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,477 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:24,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:24,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,481 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:24,485 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:24,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321444526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321444526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321444530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321444533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:24,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:24,598 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:24,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:24,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:24,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,603 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,603 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:24,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:24,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:24,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:24,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:24,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:24,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:24,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:24,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:24,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,613 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:24,613 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:24,613 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,613 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:24,613 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:24,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,615 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:24,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647bce8f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:24,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:24,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,621 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:24,621 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:24,621 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,621 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:24,621 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:24,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321444636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321444636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:24,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:24,641 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:24,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:24,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:24,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:24,641 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:24,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,644 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321444640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321444641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,644 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:24,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:24,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,667 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,667 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:24,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,668 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:24,668 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:24,768 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:24,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:24,780 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:24,780 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:24,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:24,782 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,782 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,786 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,786 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:24,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:24,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:24,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:24,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:24,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:24,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:24,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:24,821 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,822 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:24,830 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:24,830 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:24,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:24,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:24,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:24,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,833 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,834 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,834 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:24,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7319defb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:24,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:24,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:24,846 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:24,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:24,848 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:24,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321444847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321444847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321444847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,853 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:24,853 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321444855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:24,862 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:24,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:24,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:24,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:24,862 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:24,864 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,864 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:24,864 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,865 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,868 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:24,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:24,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:24,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:24,876 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:24,876 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:24,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:24,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:24,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:24,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:24,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:24,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:24,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/42ed8096c4d743bfbd422d00b261b3d1 2023-04-01 03:56:24,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/81d0f3977fad40798f73adbf2762ed93 2023-04-01 03:56:24,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:24,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:24,999 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:25,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:25,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:25,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:25,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:25,029 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-01 03:56:25,029 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:25,029 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:25,034 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:25,034 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:25,034 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,034 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:25,035 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:25,035 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,036 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-01 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:25,042 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb89c85[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:25,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:25,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:25,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:25,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:25,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:25,057 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:25,057 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:25,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:25,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:25,058 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:25,065 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,065 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,066 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321445068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,074 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:25,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,078 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:25,078 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:25,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321445155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321445156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321445157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321445163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,178 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:25,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:25,191 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:25,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:25,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:25,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:25,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:25,210 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:25,210 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c849b00[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:25,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:25,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,228 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:25,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:25,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:25,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:25,230 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,231 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:25,231 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,231 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:25,236 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:25,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:25,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:25,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:25,236 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:25,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,244 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,246 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,251 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,252 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:25,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:25,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:25,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:25,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:25,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:25,365 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:25,365 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:25,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:25,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,369 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,369 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:25,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:25,386 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:25,386 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:25,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:25,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:25,400 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,400 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:25,400 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:25,401 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:25,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:25,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:25,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:25,402 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,402 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56312fa9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:25,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:25,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:25,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:25,417 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,418 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:25,428 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:25,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:25,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:25,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:25,428 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:25,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:25,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:25,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:25,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:25,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:25,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:25,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:25,451 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:25,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:25,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:25,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/a66b2fcd7b1341b6991b5bbe1acfd22d 2023-04-01 03:56:25,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/42ed8096c4d743bfbd422d00b261b3d1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1 2023-04-01 03:56:25,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1, entries=150, sequenceid=155, filesize=11.8 K 2023-04-01 03:56:25,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/81d0f3977fad40798f73adbf2762ed93 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93 2023-04-01 03:56:25,552 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:25,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:25,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93, entries=150, sequenceid=155, filesize=11.8 K 2023-04-01 03:56:25,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/a66b2fcd7b1341b6991b5bbe1acfd22d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d 2023-04-01 03:56:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:25,578 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:25,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:25,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:25,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d, entries=150, sequenceid=155, filesize=11.8 K 2023-04-01 03:56:25,589 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:25,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:25,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d055bdf2a6c005123440ac61e55072e8 in 1172ms, sequenceid=155, compaction requested=true 2023-04-01 03:56:25,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:25,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:25,596 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:25,596 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:25,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:25,603 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:25,604 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:25,604 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:25,604 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:25,604 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:25,604 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c5677728ea984a73ab6af7136172897e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.2 K 2023-04-01 03:56:25,604 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,604 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/964f60818e044232b7c1bf4b00d06bb2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.2 K 2023-04-01 03:56:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:25,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:25,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:25,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:25,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:25,609 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:25,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:25,614 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 964f60818e044232b7c1bf4b00d06bb2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:25,614 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c5677728ea984a73ab6af7136172897e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:25,615 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting febdb947f7e140b0bedb0a9aaa842179, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680321382865 2023-04-01 03:56:25,616 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 31c3aca2c49242e3a11f4bd7c2d38ce8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680321382865 2023-04-01 03:56:25,616 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 81d0f3977fad40798f73adbf2762ed93, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:25,617 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 42ed8096c4d743bfbd422d00b261b3d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:25,626 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:25,626 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:25,627 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:25,628 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:25,655 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#20 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:25,655 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#19 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:25,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:25,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:25,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321445707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321445714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321445714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321445719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321445837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321445837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321445837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:25,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321445846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321446044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321446047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321446049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321446057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,156 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/dce98ddcf3b747dcbd6e699733e62de8 2023-04-01 03:56:26,160 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/39adf773958840818c2e2c8f780bd1d4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/39adf773958840818c2e2c8f780bd1d4 2023-04-01 03:56:26,171 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/8a2bb1f1f217415cbac30681325c4d02 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8a2bb1f1f217415cbac30681325c4d02 2023-04-01 03:56:26,175 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 39adf773958840818c2e2c8f780bd1d4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:26,176 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:26,176 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321385596; duration=0sec 2023-04-01 03:56:26,177 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:26,177 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:26,181 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:26,181 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:26,182 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,182 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e975b45d61354b17ac39ae3c93618e9e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.2 K 2023-04-01 03:56:26,184 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e975b45d61354b17ac39ae3c93618e9e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321382080 2023-04-01 03:56:26,185 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 13dc9e7bc380458ba83b43d3c5a8e215, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680321382865 2023-04-01 03:56:26,185 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a66b2fcd7b1341b6991b5bbe1acfd22d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:26,196 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 8a2bb1f1f217415cbac30681325c4d02(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:26,202 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:26,202 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321385596; duration=0sec 2023-04-01 03:56:26,203 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:26,204 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/fe8fda2cbe6b49fa8fba85bffa58edc1 2023-04-01 03:56:26,222 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#22 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:26,274 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2238ffc2c9f2481198e78dd300baefc9 2023-04-01 03:56:26,275 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/f78469fc1f184653b3dabb163fdc049a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f78469fc1f184653b3dabb163fdc049a 2023-04-01 03:56:26,285 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/dce98ddcf3b747dcbd6e699733e62de8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8 2023-04-01 03:56:26,293 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into f78469fc1f184653b3dabb163fdc049a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:26,294 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:26,294 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321385596; duration=0sec 2023-04-01 03:56:26,294 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:26,300 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8, entries=150, sequenceid=175, filesize=11.8 K 2023-04-01 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/fe8fda2cbe6b49fa8fba85bffa58edc1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1 2023-04-01 03:56:26,311 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1, entries=150, sequenceid=175, filesize=11.8 K 2023-04-01 03:56:26,315 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2238ffc2c9f2481198e78dd300baefc9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9 2023-04-01 03:56:26,322 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9, entries=150, sequenceid=175, filesize=11.8 K 2023-04-01 03:56:26,324 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d055bdf2a6c005123440ac61e55072e8 in 697ms, sequenceid=175, compaction requested=false 2023-04-01 03:56:26,324 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:26,324 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,324 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:26,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:26,327 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b62dd92[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:26,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:26,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,332 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:26,332 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:26,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:26,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:26,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:26,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:26,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:26,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:26,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:26,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:26,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,344 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,344 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:26,345 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:26,345 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,345 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:26,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:26,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321446391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321446401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321446402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321446407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,446 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:26,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:26,457 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:26,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:26,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:26,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:26,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:26,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:26,483 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,483 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:26,484 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:26,484 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:26,490 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,490 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@104538cf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:26,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:26,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,494 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,494 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:26,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:26,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:26,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:26,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:26,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:26,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:26,504 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:26,504 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:26,504 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:26,504 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:26,504 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:26,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321446507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,509 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,509 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,509 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321446511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,513 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321446515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:26,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321446515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,518 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:26,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:26,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:26,619 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:26,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:26,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:26,624 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:26,624 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,625 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:26,625 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:26,625 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,625 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,626 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,626 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:26,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:26,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:26,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:26,628 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:26,628 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:26,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:26,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:26,630 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,630 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:26,630 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:26,631 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:26,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:26,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:26,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:26,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5851fd5c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:26,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:26,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,636 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:26,637 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,637 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:26,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:26,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:26,641 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:26,645 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:26,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:26,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:26,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:26,645 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:26,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,676 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,676 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:26,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:26,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,676 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,677 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:26,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:26,677 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321446711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321446714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321446717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:26,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321446721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,777 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:26,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:26,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:26,792 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:26,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:26,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:26,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,794 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,794 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:26,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:26,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:26,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:26,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:26,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:26,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:26,799 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:26,799 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:26,799 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:26,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:26,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:26,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:26,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/5457617001f64faca57df208a40adb66 2023-04-01 03:56:26,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:26,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:26,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742dd98[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:26,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:26,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,805 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:26,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:26,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:26,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:26,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:26,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:26,806 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:26,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:26,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:26,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:26,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:26,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:26,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:26,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:26,815 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:26,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:26,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:26,819 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,819 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,819 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:26,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:26,819 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:26,819 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,820 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:26,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:26,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:26,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:26,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/97c9836351cf4734ae2330b61ee57b65 2023-04-01 03:56:26,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/5ff18ba99ede4814ba8c5523d76de39e 2023-04-01 03:56:26,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/5457617001f64faca57df208a40adb66 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66 2023-04-01 03:56:26,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66, entries=150, sequenceid=193, filesize=11.8 K 2023-04-01 03:56:26,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/97c9836351cf4734ae2330b61ee57b65 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65 2023-04-01 03:56:26,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65, entries=150, sequenceid=193, filesize=11.8 K 2023-04-01 03:56:26,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/5ff18ba99ede4814ba8c5523d76de39e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e 2023-04-01 03:56:26,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e, entries=150, sequenceid=193, filesize=11.8 K 2023-04-01 03:56:26,907 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for d055bdf2a6c005123440ac61e55072e8 in 555ms, sequenceid=193, compaction requested=true 2023-04-01 03:56:26,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:26,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:26,907 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:26,907 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:26,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:26,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:26,909 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:26,910 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:26,910 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:26,910 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,910 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:26,910 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,910 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8a2bb1f1f217415cbac30681325c4d02, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.5 K 2023-04-01 03:56:26,910 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/39adf773958840818c2e2c8f780bd1d4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.5 K 2023-04-01 03:56:26,911 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 8a2bb1f1f217415cbac30681325c4d02, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:26,911 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 39adf773958840818c2e2c8f780bd1d4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:26,913 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting dce98ddcf3b747dcbd6e699733e62de8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680321384524 2023-04-01 03:56:26,913 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fe8fda2cbe6b49fa8fba85bffa58edc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680321384524 2023-04-01 03:56:26,913 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5457617001f64faca57df208a40adb66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680321385702 2023-04-01 03:56:26,913 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 97c9836351cf4734ae2330b61ee57b65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680321385702 2023-04-01 03:56:26,920 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:26,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:26,926 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#28 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:26,926 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:26,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:26,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:26,927 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:26,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:26,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:26,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:26,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:26,934 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:26,934 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:26,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:26,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:26,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:26,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:26,956 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:26,956 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:26,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:26,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:26,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:26,962 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:56:26,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:26,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:26,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:26,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:26,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:26,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:27,021 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/dd5c7a78d32340d0b13884d0ebc440d7 2023-04-01 03:56:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321447042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321447051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321447053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321447058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,075 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b9875610403b47ae871849244fc3112d 2023-04-01 03:56:27,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321447091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,094 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:27,126 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/0970e700074744c08ae05d2c0b32d3ce 2023-04-01 03:56:27,135 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/dd5c7a78d32340d0b13884d0ebc440d7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7 2023-04-01 03:56:27,146 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7, entries=150, sequenceid=211, filesize=11.8 K 2023-04-01 03:56:27,150 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b9875610403b47ae871849244fc3112d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d 2023-04-01 03:56:27,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321447157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321447160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321447160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,164 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d, entries=150, sequenceid=211, filesize=11.8 K 2023-04-01 03:56:27,167 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/0970e700074744c08ae05d2c0b32d3ce as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce 2023-04-01 03:56:27,196 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce, entries=150, sequenceid=211, filesize=11.8 K 2023-04-01 03:56:27,198 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for d055bdf2a6c005123440ac61e55072e8 in 241ms, sequenceid=211, compaction requested=true 2023-04-01 03:56:27,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:27,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:27,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:27,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:27,203 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,203 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:27,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:56:27,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:27,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:27,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ab11eaf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:27,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:27,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,213 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,213 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,213 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,213 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:27,213 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:27,213 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:27,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:27,216 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:27,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:27,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:27,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:27,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/7e414969940a443c937c2451d37a89a8 2023-04-01 03:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,251 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,254 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,254 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:27,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:27,255 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:27,255 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:27,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:27,262 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,262 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321447314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,355 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:27,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:27,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:27,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:27,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:27,360 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:27,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:27,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:27,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:27,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321447362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321447364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321447367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:27,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:27,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:27,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:27,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:27,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:27,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,384 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:27,384 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:27,384 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:27,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:27,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:27,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:27,395 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,395 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/a0e83d4a981143a19123d4495538f32d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/a0e83d4a981143a19123d4495538f32d 2023-04-01 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,398 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,398 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,399 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/1b86764df117470691621fd17518a4b3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1b86764df117470691621fd17518a4b3 2023-04-01 03:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430bd276[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:27,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:27,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,404 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:27,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:27,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:27,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:27,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:27,405 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,414 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into a0e83d4a981143a19123d4495538f32d(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:27,415 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:27,415 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321386907; duration=0sec 2023-04-01 03:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:27,416 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:27,416 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:27,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:27,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:27,432 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:27,416 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:27,432 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:27,433 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 1b86764df117470691621fd17518a4b3(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:27,433 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,433 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321386907; duration=0sec 2023-04-01 03:56:27,433 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:27,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,438 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:27,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,439 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:27,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,439 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,439 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f78469fc1f184653b3dabb163fdc049a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=47.2 K 2023-04-01 03:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f78469fc1f184653b3dabb163fdc049a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321384347 2023-04-01 03:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2238ffc2c9f2481198e78dd300baefc9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680321384524 2023-04-01 03:56:27,450 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5ff18ba99ede4814ba8c5523d76de39e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680321385702 2023-04-01 03:56:27,451 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0970e700074744c08ae05d2c0b32d3ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680321386390 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,452 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:27,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:27,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:27,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:27,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321447434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,473 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#34 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:27,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:27,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:27,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:27,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:27,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:27,562 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:27,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:27,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:27,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,564 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:27,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:27,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:27,566 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:27,566 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,566 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:27,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:27,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:27,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:27,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:27,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:27,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,573 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14677c56[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:27,574 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:27,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,576 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:27,576 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:27,576 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:27,576 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,576 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:27,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:27,579 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:27,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:27,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:27,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:27,580 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:27,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,582 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,583 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,588 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,588 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,588 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,592 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,592 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:27,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:27,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:27,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,592 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:27,592 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:27,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321447656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321447668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321447668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321447672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/0386015aaa364128bb6d4930717f85e1 2023-04-01 03:56:27,693 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:27,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:27,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:27,702 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:27,702 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,702 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:27,702 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:27,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,707 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,708 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:27,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:27,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:27,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/c936130a200b43118ad2c6e69993bffe 2023-04-01 03:56:27,745 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,745 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:27,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/7e414969940a443c937c2451d37a89a8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8 2023-04-01 03:56:27,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d8dd5f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:27,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:27,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,767 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:27,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:27,767 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:27,767 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:27,768 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:27,768 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,768 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:27,769 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:27,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:27,773 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:27,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:27,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:27,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:27,773 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:27,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8, entries=200, sequenceid=227, filesize=14.1 K 2023-04-01 03:56:27,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/0386015aaa364128bb6d4930717f85e1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1 2023-04-01 03:56:27,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1, entries=150, sequenceid=227, filesize=11.8 K 2023-04-01 03:56:27,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/c936130a200b43118ad2c6e69993bffe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe 2023-04-01 03:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:27,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:27,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:27,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:27,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe, entries=150, sequenceid=227, filesize=11.8 K 2023-04-01 03:56:27,816 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d055bdf2a6c005123440ac61e55072e8 in 612ms, sequenceid=227, compaction requested=true 2023-04-01 03:56:27,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:27,816 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:27,818 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:27,818 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:27,818 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,819 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/a0e83d4a981143a19123d4495538f32d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=37.9 K 2023-04-01 03:56:27,819 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a0e83d4a981143a19123d4495538f32d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680321385702 2023-04-01 03:56:27,820 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting dd5c7a78d32340d0b13884d0ebc440d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680321386390 2023-04-01 03:56:27,820 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7e414969940a443c937c2451d37a89a8, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680321387026 2023-04-01 03:56:27,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:27,823 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 03:56:27,835 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:27,889 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/143519f9415a4eb0b8817d498372b527 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/143519f9415a4eb0b8817d498372b527 2023-04-01 03:56:27,898 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into 143519f9415a4eb0b8817d498372b527(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:27,898 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,898 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=12, startTime=1680321386907; duration=0sec 2023-04-01 03:56:27,898 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 03:56:27,898 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:27,900 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:27,901 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:27,901 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,901 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1b86764df117470691621fd17518a4b3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.6 K 2023-04-01 03:56:27,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:27,902 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1b86764df117470691621fd17518a4b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680321385702 2023-04-01 03:56:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:27,903 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b9875610403b47ae871849244fc3112d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680321386390 2023-04-01 03:56:27,904 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0386015aaa364128bb6d4930717f85e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680321387026 2023-04-01 03:56:27,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:27,926 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:27,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:27,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:27,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,932 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:27,954 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,954 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:27,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:27,955 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,955 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:27,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:27,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:27,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:27,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:27,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:27,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:27,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:27,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:27,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:27,973 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/1eefbc00d9564bb79e4f312bcaf2784f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1eefbc00d9564bb79e4f312bcaf2784f 2023-04-01 03:56:27,975 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:27,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@776eef2d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:27,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:27,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:27,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:27,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:27,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:27,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:27,986 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,987 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,987 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:27,987 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 1eefbc00d9564bb79e4f312bcaf2784f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:27,987 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,987 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321387816; duration=0sec 2023-04-01 03:56:27,987 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:27,987 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 03:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,989 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 03:56:27,989 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 03:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,989 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. because compaction request was cancelled 2023-04-01 03:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:27,990 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:27,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:27,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:27,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:27,991 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:27,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:27,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:27,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:27,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:27,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321448017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,104 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,107 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,110 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,110 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,112 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,112 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,112 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,112 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,113 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,114 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,114 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aeb1bd4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,121 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,121 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,121 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,121 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,121 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321448121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,129 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321448173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321448174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321448179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,247 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,248 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,248 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,249 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,249 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,249 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,249 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,251 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,251 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,251 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,251 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,252 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c15f13[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,253 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,253 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,254 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,257 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,257 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,261 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,262 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,262 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,264 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,264 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,275 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/5580839f0533471da6cde5db00d2ea0a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5580839f0533471da6cde5db00d2ea0a 2023-04-01 03:56:28,285 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 5580839f0533471da6cde5db00d2ea0a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:28,285 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:28,285 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321387816; duration=0sec 2023-04-01 03:56:28,285 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:28,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321448327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,364 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,368 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,382 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,382 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,382 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,382 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,383 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b050d1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,387 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,387 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,387 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,388 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,389 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,390 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,391 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,392 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,392 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,392 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,393 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,394 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:28,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/79a3b5c5a5d643e4b3f58b96c077be40 2023-04-01 03:56:28,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,501 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,502 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,506 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,507 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,507 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,510 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,514 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,514 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,514 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,515 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,516 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,516 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a7d0ad[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,519 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,527 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,527 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,528 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,528 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,528 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,548 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,548 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,549 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,549 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:28,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43502 deadline: 1680321448633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,649 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,654 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,656 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a04e16[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,695 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,695 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,695 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,699 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,699 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,701 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,701 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,701 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,705 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,705 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,706 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,706 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,706 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,706 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,706 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,806 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,811 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,811 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,814 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,814 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,814 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:28,814 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:28,814 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:28,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/52b91a7352364655a56b906d21348c5f 2023-04-01 03:56:28,832 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,832 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,832 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,832 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b56a656[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:28,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:28,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:28,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:28,841 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:28,844 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:28,844 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:28,844 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:28,844 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:28,845 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:28,846 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,846 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:28,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/e2ea74deb9e1494b93a735ab1d7860b8 2023-04-01 03:56:28,846 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:28,847 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,849 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/79a3b5c5a5d643e4b3f58b96c077be40 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40 2023-04-01 03:56:28,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:28,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:28,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:28,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:28,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:28,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:28,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40, entries=200, sequenceid=247, filesize=14.1 K 2023-04-01 03:56:28,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/52b91a7352364655a56b906d21348c5f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f 2023-04-01 03:56:28,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f, entries=150, sequenceid=247, filesize=11.8 K 2023-04-01 03:56:28,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/e2ea74deb9e1494b93a735ab1d7860b8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8 2023-04-01 03:56:28,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8, entries=150, sequenceid=247, filesize=11.8 K 2023-04-01 03:56:28,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d055bdf2a6c005123440ac61e55072e8 in 911ms, sequenceid=247, compaction requested=true 2023-04-01 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:28,873 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 03:56:28,874 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 03:56:28,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:28,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:28,874 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 03:56:28,874 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 03:56:28,874 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 03:56:28,875 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. because compaction request was cancelled 2023-04-01 03:56:28,875 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 03:56:28,875 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:28,875 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. because compaction request was cancelled 2023-04-01 03:56:28,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:28,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:28,876 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,876 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/143519f9415a4eb0b8817d498372b527, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=35.6 K 2023-04-01 03:56:28,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 143519f9415a4eb0b8817d498372b527, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1680321386390 2023-04-01 03:56:28,877 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c936130a200b43118ad2c6e69993bffe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680321387026 2023-04-01 03:56:28,877 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e2ea74deb9e1494b93a735ab1d7860b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680321387298 2023-04-01 03:56:28,884 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#41 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:28,959 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:28,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:28,962 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:28,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:28,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:28,963 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,963 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,965 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:28,965 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:28,965 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:28,965 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:28,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:28,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:28,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:28,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:28,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:28,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:28,968 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:56:28,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:28,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:28,969 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:28,969 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:28,969 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:28,969 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:28,979 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/4dc8e54b3e3e4173a1f5ff79d57b767c 2023-04-01 03:56:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:29,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:29,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321449184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321449187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,300 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/6697c32ecb30449c9305c8d9b0442b49 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6697c32ecb30449c9305c8d9b0442b49 2023-04-01 03:56:29,307 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into 6697c32ecb30449c9305c8d9b0442b49(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:29,307 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:29,307 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321388874; duration=0sec 2023-04-01 03:56:29,307 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:29,392 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/8aaffd4f2d19438b980e041642060117 2023-04-01 03:56:29,405 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/8a2ce1435e774fd2ba3e2baa73b48daa 2023-04-01 03:56:29,414 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/4dc8e54b3e3e4173a1f5ff79d57b767c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c 2023-04-01 03:56:29,422 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c, entries=150, sequenceid=264, filesize=11.9 K 2023-04-01 03:56:29,423 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/8aaffd4f2d19438b980e041642060117 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117 2023-04-01 03:56:29,439 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117, entries=150, sequenceid=264, filesize=11.9 K 2023-04-01 03:56:29,440 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/8a2ce1435e774fd2ba3e2baa73b48daa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa 2023-04-01 03:56:29,447 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa, entries=150, sequenceid=264, filesize=11.9 K 2023-04-01 03:56:29,448 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for d055bdf2a6c005123440ac61e55072e8 in 480ms, sequenceid=264, compaction requested=true 2023-04-01 03:56:29,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:29,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:29,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f644735[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:29,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:29,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,458 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:29,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:29,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:29,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:29,459 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:29,462 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:29,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:29,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:29,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:29,462 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:29,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:29,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:29,464 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:29,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:29,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:29,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:29,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:29,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:29,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:29,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:29,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:29,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:29,486 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/c854e8adb91a4db8927ef3391db3c4b0 2023-04-01 03:56:29,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/67cbd2bc04494d23bdb2177865a7a2c4 2023-04-01 03:56:29,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/441644d65868475ea746645bb0cd9fe3 2023-04-01 03:56:29,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/c854e8adb91a4db8927ef3391db3c4b0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0 2023-04-01 03:56:29,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0, entries=200, sequenceid=284, filesize=14.3 K 2023-04-01 03:56:29,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/67cbd2bc04494d23bdb2177865a7a2c4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4 2023-04-01 03:56:29,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4, entries=150, sequenceid=284, filesize=11.9 K 2023-04-01 03:56:29,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/441644d65868475ea746645bb0cd9fe3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3 2023-04-01 03:56:29,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3, entries=150, sequenceid=284, filesize=11.9 K 2023-04-01 03:56:29,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d055bdf2a6c005123440ac61e55072e8 in 67ms, sequenceid=284, compaction requested=true 2023-04-01 03:56:29,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:29,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:29,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:29,554 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:29,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:29,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:29,555 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53684 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:29,555 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48854 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:29,555 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:29,555 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:29,555 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:29,555 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:29,556 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1eefbc00d9564bb79e4f312bcaf2784f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=47.7 K 2023-04-01 03:56:29,556 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5580839f0533471da6cde5db00d2ea0a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=52.4 K 2023-04-01 03:56:29,556 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1eefbc00d9564bb79e4f312bcaf2784f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680321387026 2023-04-01 03:56:29,556 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5580839f0533471da6cde5db00d2ea0a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680321387026 2023-04-01 03:56:29,556 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 52b91a7352364655a56b906d21348c5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680321387298 2023-04-01 03:56:29,556 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 79a3b5c5a5d643e4b3f58b96c077be40, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680321387298 2023-04-01 03:56:29,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8aaffd4f2d19438b980e041642060117, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680321388001 2023-04-01 03:56:29,557 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4dc8e54b3e3e4173a1f5ff79d57b767c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680321388001 2023-04-01 03:56:29,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 67cbd2bc04494d23bdb2177865a7a2c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389179 2023-04-01 03:56:29,557 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c854e8adb91a4db8927ef3391db3c4b0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389172 2023-04-01 03:56:29,566 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:29,566 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:29,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:29,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:29,579 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:29,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:29,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:29,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:29,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,579 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:29,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:29,581 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:29,582 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,582 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:29,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:29,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:29,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:29,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:29,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:29,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:29,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:29,591 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/7f1faed54b804db8be9e7c3121ab1db3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7f1faed54b804db8be9e7c3121ab1db3 2023-04-01 03:56:29,594 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:29,594 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:29,594 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:29,599 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 7f1faed54b804db8be9e7c3121ab1db3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:29,599 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:29,599 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=12, startTime=1680321389553; duration=0sec 2023-04-01 03:56:29,599 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:29,599 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:29,602 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:29,602 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:29,602 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:29,603 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6697c32ecb30449c9305c8d9b0442b49, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.0 K 2023-04-01 03:56:29,608 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6697c32ecb30449c9305c8d9b0442b49, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680321387298 2023-04-01 03:56:29,608 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 8a2ce1435e774fd2ba3e2baa73b48daa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680321388001 2023-04-01 03:56:29,609 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 441644d65868475ea746645bb0cd9fe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389179 2023-04-01 03:56:29,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:29,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:29,619 WARN [DataStreamer for file /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e65a491314b9473a990729bfff2d4381] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 03:56:29,619 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e65a491314b9473a990729bfff2d4381 2023-04-01 03:56:29,631 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:29,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321449957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:29,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321449957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:29,993 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/94341de8cc654de8a146f051a7eb4a75 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/94341de8cc654de8a146f051a7eb4a75 2023-04-01 03:56:30,001 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 94341de8cc654de8a146f051a7eb4a75(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:30,001 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:30,002 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=12, startTime=1680321389554; duration=0sec 2023-04-01 03:56:30,002 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:30,057 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/4cd69964dbb942edb0f0e7b42baf1fd4 2023-04-01 03:56:30,076 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/75c5ecb2545f4188a4775a6dfb3b94a5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/75c5ecb2545f4188a4775a6dfb3b94a5 2023-04-01 03:56:30,084 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into 75c5ecb2545f4188a4775a6dfb3b94a5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:30,084 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:30,084 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321389554; duration=0sec 2023-04-01 03:56:30,085 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:30,104 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8 2023-04-01 03:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e65a491314b9473a990729bfff2d4381 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381 2023-04-01 03:56:30,119 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381, entries=150, sequenceid=297, filesize=11.9 K 2023-04-01 03:56:30,121 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/4cd69964dbb942edb0f0e7b42baf1fd4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4 2023-04-01 03:56:30,127 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4, entries=150, sequenceid=297, filesize=11.9 K 2023-04-01 03:56:30,128 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8 2023-04-01 03:56:30,143 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8, entries=150, sequenceid=297, filesize=11.9 K 2023-04-01 03:56:30,144 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d055bdf2a6c005123440ac61e55072e8 in 550ms, sequenceid=297, compaction requested=false 2023-04-01 03:56:30,145 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:30,145 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47fed2cc[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,148 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,148 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,149 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,149 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,149 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,149 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,150 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,151 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,151 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,151 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,154 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,154 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,162 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,162 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,262 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:30,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:56:30,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:30,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:30,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:30,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:30,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:30,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,267 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,275 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,275 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,275 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,277 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,277 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,277 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,278 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,278 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,278 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/56a7eb07aaa840e792bea91521ca4bec 2023-04-01 03:56:30,280 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,281 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,281 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321450281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7198e4db[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321450282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,285 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,287 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,287 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,287 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,290 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,290 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,299 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,299 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,299 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321450383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321450385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,402 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,403 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,403 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,404 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,409 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f3298a8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,413 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,413 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,413 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,413 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,421 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,421 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,422 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,422 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,422 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,423 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,424 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,426 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:30,427 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,527 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,530 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62323d94[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,545 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,545 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:30,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,553 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,553 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321450585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321450589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,653 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,658 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,669 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536bfbac[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,672 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,676 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,676 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,676 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,676 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,676 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,679 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,679 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,681 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,683 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:30,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,686 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,686 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/926ba8e3356a4b27bad55769a6e32909 2023-04-01 03:56:30,786 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,790 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,790 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,790 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,790 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,798 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,798 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,798 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,798 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,817 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,817 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,817 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65636326[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,819 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,819 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,819 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,819 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,819 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,819 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,819 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,820 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,820 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,820 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,820 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,822 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,822 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,822 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,822 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,822 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,824 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,824 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,824 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,825 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321450889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:30,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321450894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,927 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:30,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:30,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:30,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:30,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:30,931 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:30,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:30,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:30,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:30,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:30,934 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,934 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:30,934 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:30,934 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:30,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,935 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:30,935 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:30,936 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,936 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,936 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,936 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:30,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ebf40fa[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:30,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,940 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:30,940 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:30,940 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:30,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:30,941 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:30,943 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:30,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:30,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:30,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:30,943 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:30,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:30,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,945 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:30,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,945 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:30,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:30,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:30,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:30,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:30,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:30,949 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:30,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:30,949 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:30,949 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:30,949 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:30,949 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,949 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:30,949 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,049 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:31,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:31,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:31,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:31,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:31,052 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:31,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:31,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:31,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:31,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:31,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:31,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:31,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:31,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:31,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:31,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:31,058 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:31,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d120a24[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:31,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:31,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:31,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:31,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:31,062 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,062 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:31,063 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:31,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:31,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:31,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:31,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:31,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:31,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:31,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:31,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:31,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,067 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,069 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:31,073 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:31,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:31,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:31,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,073 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:31,073 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:31,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/91a3b3077ad74627bcad7306f99507ff 2023-04-01 03:56:31,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/56a7eb07aaa840e792bea91521ca4bec as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec 2023-04-01 03:56:31,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321451127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,129 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:31,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec, entries=200, sequenceid=320, filesize=14.3 K 2023-04-01 03:56:31,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/926ba8e3356a4b27bad55769a6e32909 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909 2023-04-01 03:56:31,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909, entries=150, sequenceid=320, filesize=11.9 K 2023-04-01 03:56:31,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/91a3b3077ad74627bcad7306f99507ff as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff 2023-04-01 03:56:31,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff, entries=150, sequenceid=320, filesize=11.9 K 2023-04-01 03:56:31,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d055bdf2a6c005123440ac61e55072e8 in 880ms, sequenceid=320, compaction requested=true 2023-04-01 03:56:31,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:31,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:31,144 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:31,144 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:31,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:31,145 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:31,146 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:31,146 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:31,146 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:31,146 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,146 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:31,146 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,146 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/94341de8cc654de8a146f051a7eb4a75, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.2 K 2023-04-01 03:56:31,146 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7f1faed54b804db8be9e7c3121ab1db3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=38.6 K 2023-04-01 03:56:31,146 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 94341de8cc654de8a146f051a7eb4a75, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389179 2023-04-01 03:56:31,147 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7f1faed54b804db8be9e7c3121ab1db3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389179 2023-04-01 03:56:31,147 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4cd69964dbb942edb0f0e7b42baf1fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680321389494 2023-04-01 03:56:31,147 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e65a491314b9473a990729bfff2d4381, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680321389494 2023-04-01 03:56:31,147 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 926ba8e3356a4b27bad55769a6e32909, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389645 2023-04-01 03:56:31,147 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 56a7eb07aaa840e792bea91521ca4bec, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389636 2023-04-01 03:56:31,155 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:31,155 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:31,173 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:31,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:31,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:31,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:31,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:31,190 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:31,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:31,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:31,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:31,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:31,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:31,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:31,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:31,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:31,194 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,194 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:31,194 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:56:31,194 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:31,194 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,194 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:31,195 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,195 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:31,195 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:31,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:31,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321451232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321451235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,239 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/b3d54ddcb07442619760bb457f44a92e 2023-04-01 03:56:31,260 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b5404f09bd0d4375971f8d4c4b508930 2023-04-01 03:56:31,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321451338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321451338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321451394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321451401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321451543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321451544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,594 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/0b5f1cc8fd804faebfa1b26e43469946 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/0b5f1cc8fd804faebfa1b26e43469946 2023-04-01 03:56:31,596 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/9a405389d3d348588e379691fb773c88 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/9a405389d3d348588e379691fb773c88 2023-04-01 03:56:31,603 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 0b5f1cc8fd804faebfa1b26e43469946(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:31,603 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:31,603 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321391144; duration=0sec 2023-04-01 03:56:31,603 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:31,603 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:31,605 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:31,605 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:31,605 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,605 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/75c5ecb2545f4188a4775a6dfb3b94a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.2 K 2023-04-01 03:56:31,606 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 75c5ecb2545f4188a4775a6dfb3b94a5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680321389179 2023-04-01 03:56:31,606 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 9a405389d3d348588e379691fb773c88(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:31,606 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:31,606 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2d5b6d62ba9747b9bfd0ab52eaeae5f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680321389494 2023-04-01 03:56:31,606 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321391144; duration=0sec 2023-04-01 03:56:31,606 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:31,606 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 91a3b3077ad74627bcad7306f99507ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389645 2023-04-01 03:56:31,616 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#62 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:31,678 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/ae177a8de0e7451884c90dc1136911bc 2023-04-01 03:56:31,687 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/b3d54ddcb07442619760bb457f44a92e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e 2023-04-01 03:56:31,693 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e, entries=150, sequenceid=332, filesize=11.9 K 2023-04-01 03:56:31,694 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b5404f09bd0d4375971f8d4c4b508930 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930 2023-04-01 03:56:31,700 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930, entries=150, sequenceid=332, filesize=11.9 K 2023-04-01 03:56:31,702 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/ae177a8de0e7451884c90dc1136911bc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc 2023-04-01 03:56:31,708 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc, entries=150, sequenceid=332, filesize=11.9 K 2023-04-01 03:56:31,708 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d055bdf2a6c005123440ac61e55072e8 in 514ms, sequenceid=332, compaction requested=false 2023-04-01 03:56:31,709 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:31,709 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,709 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:31,709 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:31,709 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,709 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:31,709 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:31,711 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312820b1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:31,712 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:31,712 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,713 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:31,713 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,713 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:31,714 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:31,716 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:31,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:31,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:31,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:31,717 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:31,718 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,718 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,718 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:31,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:31,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:31,723 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:31,723 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:31,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:31,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:31,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:31,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:31,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:31,826 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:31,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:31,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:31,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,844 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:31,844 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:31,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:31,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:31,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:31,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:31,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:31,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:31,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:31,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:31,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:31,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:31,846 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:56:31,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:31,848 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,848 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:31,848 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,849 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:31,849 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:31,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:31,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:31,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321451865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321451866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321451969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:31,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:31,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321451976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,056 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/e609ca41bc0e4c78bde1ac291bb0d27c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e609ca41bc0e4c78bde1ac291bb0d27c 2023-04-01 03:56:32,066 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into e609ca41bc0e4c78bde1ac291bb0d27c(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:32,066 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:32,066 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321391145; duration=0sec 2023-04-01 03:56:32,066 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:32,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321452172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321452178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,271 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/8016c39f00e54d689a604884dba4db87 2023-04-01 03:56:32,287 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/48fa4119f47146f5bc9894151efa7406 2023-04-01 03:56:32,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43544 deadline: 1680321452406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321452412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321452476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:32,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321452481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,699 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/08a2f9bfc18e4252a6c1214e55299926 2023-04-01 03:56:32,706 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/8016c39f00e54d689a604884dba4db87 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87 2023-04-01 03:56:32,717 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87, entries=150, sequenceid=354, filesize=11.9 K 2023-04-01 03:56:32,718 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/48fa4119f47146f5bc9894151efa7406 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406 2023-04-01 03:56:32,724 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406, entries=150, sequenceid=354, filesize=11.9 K 2023-04-01 03:56:32,725 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/08a2f9bfc18e4252a6c1214e55299926 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926 2023-04-01 03:56:32,730 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926, entries=150, sequenceid=354, filesize=11.9 K 2023-04-01 03:56:32,731 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d055bdf2a6c005123440ac61e55072e8 in 885ms, sequenceid=354, compaction requested=true 2023-04-01 03:56:32,731 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:32,731 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:32,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:32,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:32,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:32,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:32,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2c1fe1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:32,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:32,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,737 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:32,737 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:32,737 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:32,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:32,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:32,739 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:32,742 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:32,742 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:32,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:32,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:32,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:32,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,745 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:32,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:32,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:32,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:32,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:32,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:32,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:32,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:32,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:32,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:32,763 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:32,763 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:32,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:32,863 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:32,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:32,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:32,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:32,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:32,866 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:32,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:32,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:32,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:32,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:32,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:32,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:32,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:32,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:32,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:32,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:32,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:32,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:32,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:32,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:32,883 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:32,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:32,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:32,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:32,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:32,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:33,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321453012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321453013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321453116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321453116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,294 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/cecc23cc24ea42b7b63a756c2c488ad8 2023-04-01 03:56:33,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321453319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321453320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43514 deadline: 1680321453623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43566 deadline: 1680321453624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,710 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/101c788d681b43e8a4cef5807c5384fa 2023-04-01 03:56:33,729 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/bb22d892bc6e4134bb60a06a9edafefe 2023-04-01 03:56:33,733 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/cecc23cc24ea42b7b63a756c2c488ad8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8 2023-04-01 03:56:33,739 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:56:33,740 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/101c788d681b43e8a4cef5807c5384fa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa 2023-04-01 03:56:33,748 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:56:33,749 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/bb22d892bc6e4134bb60a06a9edafefe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe 2023-04-01 03:56:33,754 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:56:33,755 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d055bdf2a6c005123440ac61e55072e8 in 872ms, sequenceid=367, compaction requested=true 2023-04-01 03:56:33,755 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:33,755 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:33,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:33,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:33,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:33,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:33,757 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,757 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:33,757 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:33,757 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b202e6e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:33,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:33,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:33,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:33,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:33,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:33,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:33,765 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:33,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:33,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:33,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:33,765 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:33,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:33,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,768 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:33,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:33,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:33,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:33,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:33,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:33,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:33,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:33,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:33,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:33,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:33,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:33,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:33,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:33,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:33,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:33,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:33,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:33,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:33,877 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:33,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:33,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:33,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:33,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:33,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:33,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:33,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:33,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:33,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:33,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:33,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:33,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:33,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:33,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:33,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:33,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:33,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:33,881 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:33,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:34,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:34,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:34,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321454144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321454144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321454245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321454249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,291 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/534547d616f1466b911caad61959a4ff 2023-04-01 03:56:34,303 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b385cf79096a442cadee5deb781c8a51 2023-04-01 03:56:34,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43544 deadline: 1680321454420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,423 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:34,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321454430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,432 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:34,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321454447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321454453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,715 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/89ef8a4cdedc451a8c0d1d8e023684bf 2023-04-01 03:56:34,720 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/534547d616f1466b911caad61959a4ff as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff 2023-04-01 03:56:34,729 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff, entries=150, sequenceid=387, filesize=11.9 K 2023-04-01 03:56:34,730 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/b385cf79096a442cadee5deb781c8a51 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51 2023-04-01 03:56:34,736 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51, entries=150, sequenceid=387, filesize=11.9 K 2023-04-01 03:56:34,736 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/89ef8a4cdedc451a8c0d1d8e023684bf as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf 2023-04-01 03:56:34,741 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf, entries=150, sequenceid=387, filesize=11.9 K 2023-04-01 03:56:34,742 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d055bdf2a6c005123440ac61e55072e8 in 861ms, sequenceid=387, compaction requested=true 2023-04-01 03:56:34,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:34,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:34,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:34,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:34,746 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ce4804[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:34,748 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:34,748 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:34,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:34,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,750 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,750 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:34,751 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:34,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:34,753 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:34,753 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:34,753 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:34,753 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:34,753 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:34,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:56:34,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:34,756 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:34,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:34,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:34,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:34,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:34,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:34,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:34,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:34,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:34,764 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:34,764 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321454784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321454786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,864 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:34,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:34,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:34,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:34,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:34,867 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:34,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:34,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:34,867 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:34,868 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,868 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,869 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:34,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:34,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:34,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:34,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:34,870 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:34,870 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:34,870 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:34,870 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:34,870 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:34,870 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:34,871 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:34,871 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:34,871 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:34,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:34,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:34,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:34,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:34,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,872 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,872 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,872 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:34,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2613d7c4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:34,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:34,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:34,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:34,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:34,874 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,874 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:34,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,875 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,875 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,875 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:34,877 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:34,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:34,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:34,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:34,877 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:34,878 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,878 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:34,878 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:34,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:34,883 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:34,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:34,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,883 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:34,883 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:34,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321454888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:34,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321454888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:34,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:34,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:34,986 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:34,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:34,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:34,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:34,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,988 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:34,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:34,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:34,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:34,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:34,989 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:34,989 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:34,989 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:34,989 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:34,989 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:34,989 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:34,989 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:34,989 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:34,990 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:34,990 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:34,990 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:34,990 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,990 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:34,990 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:34,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:34,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96f17f0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:34,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:34,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,993 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:34,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:34,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:34,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:34,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:34,994 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,995 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:34,995 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:34,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:34,997 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:34,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:34,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:34,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:34,997 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:34,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:34,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:34,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,001 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,001 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:35,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321455091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321455092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,101 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,104 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,107 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,108 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,108 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,108 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,108 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,108 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,108 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,108 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,109 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,109 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:35,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73460bed[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:35,112 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:35,112 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,114 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,114 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:35,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:35,114 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:35,114 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:35,115 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,116 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:35,116 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,116 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:35,117 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:35,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:35,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:35,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:35,118 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:35,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:35,122 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:35,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/f7f44fc69f264abca62512f47fa24410 2023-04-01 03:56:35,223 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,226 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,227 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,230 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,230 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,230 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,230 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:35,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:35,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:35,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576cf3e9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:35,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:35,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:35,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:35,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:35,237 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,237 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:35,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:35,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:35,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:35,240 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:35,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:35,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:35,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:35,240 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:35,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,242 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,244 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,347 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,349 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:35,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:35,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,352 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f79a337[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:35,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:35,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,355 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:35,355 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:35,355 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:35,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:35,356 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:35,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:35,358 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:35,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:35,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:35,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:35,358 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:35,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,364 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321455396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321455397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,464 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,467 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,470 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:35,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:35,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e58f9bf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:35,478 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:35,478 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,480 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:35,480 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:35,480 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:35,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:35,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:35,486 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:35,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:35,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:35,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:35,486 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:35,488 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,488 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,492 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,496 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,496 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/25c3c933fdad4df898042c0c4139bd65 2023-04-01 03:56:35,596 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,600 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,603 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,603 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,605 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,605 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,605 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/db8e103a4a284b1797aa285ac330d02d 2023-04-01 03:56:35,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:35,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:35,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:35,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:35,614 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:35,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/f7f44fc69f264abca62512f47fa24410 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410 2023-04-01 03:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:35,619 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:35,619 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d8280fa[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:35,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:35,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:35,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:35,623 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,623 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:35,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410, entries=200, sequenceid=400, filesize=14.3 K 2023-04-01 03:56:35,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:35,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/25c3c933fdad4df898042c0c4139bd65 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65 2023-04-01 03:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:35,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:35,627 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:35,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:35,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:35,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:35,627 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:35,628 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,628 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,628 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:35,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:35,632 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:35,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65, entries=150, sequenceid=400, filesize=11.9 K 2023-04-01 03:56:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,634 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,634 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,634 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,634 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,634 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:35,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:35,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:35,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:35,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:35,635 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:35,635 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:35,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:35,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:35,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/db8e103a4a284b1797aa285ac330d02d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d 2023-04-01 03:56:35,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d, entries=150, sequenceid=400, filesize=11.9 K 2023-04-01 03:56:35,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d055bdf2a6c005123440ac61e55072e8 in 889ms, sequenceid=400, compaction requested=true 2023-04-01 03:56:35,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:35,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:35,645 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:56:35,645 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:56:35,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:35,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:35,647 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:56:35,647 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:35,647 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,648 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/9a405389d3d348588e379691fb773c88, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=72.0 K 2023-04-01 03:56:35,648 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9a405389d3d348588e379691fb773c88, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389645 2023-04-01 03:56:35,648 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76204 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:56:35,648 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:35,648 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,648 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/0b5f1cc8fd804faebfa1b26e43469946, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=74.4 K 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b5404f09bd0d4375971f8d4c4b508930, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680321390272 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0b5f1cc8fd804faebfa1b26e43469946, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389645 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 48fa4119f47146f5bc9894151efa7406, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680321391229 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b3d54ddcb07442619760bb457f44a92e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680321390272 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 101c788d681b43e8a4cef5807c5384fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321391855 2023-04-01 03:56:35,649 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 8016c39f00e54d689a604884dba4db87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680321391229 2023-04-01 03:56:35,650 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b385cf79096a442cadee5deb781c8a51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680321393007 2023-04-01 03:56:35,650 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting cecc23cc24ea42b7b63a756c2c488ad8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321391855 2023-04-01 03:56:35,650 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 25c3c933fdad4df898042c0c4139bd65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:35,650 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 534547d616f1466b911caad61959a4ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680321393007 2023-04-01 03:56:35,651 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f7f44fc69f264abca62512f47fa24410, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:35,668 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:35,676 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:35,735 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:35,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:35,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:35,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:35,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:35,738 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:35,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:35,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:35,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,756 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:35,756 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:35,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:35,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:35,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:35,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:35,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:35,758 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:35,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:35,765 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/3b2f8d16bc68432f8777ba9435488854 2023-04-01 03:56:35,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:35,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:35,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321455919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:35,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:35,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321455921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321456022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321456024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,089 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/f1949e0d6549428aa0bba1ee026be679 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/f1949e0d6549428aa0bba1ee026be679 2023-04-01 03:56:36,095 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into f1949e0d6549428aa0bba1ee026be679(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:36,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:36,095 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=10, startTime=1680321395645; duration=0sec 2023-04-01 03:56:36,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:36,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:56:36,097 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:56:36,097 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/04304e0606614882a81b119402e85191 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/04304e0606614882a81b119402e85191 2023-04-01 03:56:36,097 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:36,097 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,098 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e609ca41bc0e4c78bde1ac291bb0d27c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=72.0 K 2023-04-01 03:56:36,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e609ca41bc0e4c78bde1ac291bb0d27c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321389645 2023-04-01 03:56:36,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ae177a8de0e7451884c90dc1136911bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680321390272 2023-04-01 03:56:36,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 08a2f9bfc18e4252a6c1214e55299926, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680321391229 2023-04-01 03:56:36,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bb22d892bc6e4134bb60a06a9edafefe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321391855 2023-04-01 03:56:36,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 89ef8a4cdedc451a8c0d1d8e023684bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680321393007 2023-04-01 03:56:36,100 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting db8e103a4a284b1797aa285ac330d02d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:36,108 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 04304e0606614882a81b119402e85191(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:36,108 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:36,108 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=10, startTime=1680321395645; duration=0sec 2023-04-01 03:56:36,108 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:36,115 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:36,190 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/29aeb66d164148399da76864dc5c2b3f 2023-04-01 03:56:36,200 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/946c04f879bc4bc994286fe799e5d724 2023-04-01 03:56:36,205 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/3b2f8d16bc68432f8777ba9435488854 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854 2023-04-01 03:56:36,210 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854, entries=150, sequenceid=419, filesize=11.9 K 2023-04-01 03:56:36,211 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/29aeb66d164148399da76864dc5c2b3f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f 2023-04-01 03:56:36,216 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f, entries=150, sequenceid=419, filesize=11.9 K 2023-04-01 03:56:36,217 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/946c04f879bc4bc994286fe799e5d724 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724 2023-04-01 03:56:36,221 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724, entries=150, sequenceid=419, filesize=11.9 K 2023-04-01 03:56:36,222 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d055bdf2a6c005123440ac61e55072e8 in 464ms, sequenceid=419, compaction requested=false 2023-04-01 03:56:36,223 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:36,223 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,224 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d41d318[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:36,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:56:36,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:36,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:36,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:36,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:36,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:36,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:36,228 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,232 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,232 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,233 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,234 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,234 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,238 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,239 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,239 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,239 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,239 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/6d7c5be4fdac43ddbe536e106d125f15 2023-04-01 03:56:36,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321456285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321456285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,339 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,342 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,347 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f03b59c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,351 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,353 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,353 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,353 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,353 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,353 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,358 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321456390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321456390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,458 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,461 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,462 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164f38c2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,487 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,487 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,487 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,488 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,488 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,490 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,491 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,492 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,497 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,497 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,497 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,529 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/1c5366585b374ca0a30865df974f8283 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1c5366585b374ca0a30865df974f8283 2023-04-01 03:56:36,536 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into 1c5366585b374ca0a30865df974f8283(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:36,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:36,536 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=10, startTime=1680321395646; duration=0sec 2023-04-01 03:56:36,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:36,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321456593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321456594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,599 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,603 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,603 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,603 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,604 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f03efa1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,608 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,609 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,609 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,609 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,611 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,612 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,613 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,613 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:36,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,619 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,619 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/6303cb0749b04a5787ee439b7e07dba1 2023-04-01 03:56:36,719 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,722 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,724 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,725 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,726 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,726 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,726 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,726 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,728 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c393e4a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,731 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,736 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,736 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,740 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,743 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,843 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,846 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,848 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,848 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,848 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,849 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,852 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e44c304[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,854 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,855 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,856 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,857 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,857 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,858 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,859 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,861 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:36,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321456896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:36,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321456897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,961 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:36,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:36,964 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:36,964 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,964 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:36,964 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:36,964 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,964 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:36,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:36,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:36,967 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:36,967 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:36,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:36,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:36,968 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:36,968 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:36,969 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,969 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,970 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,970 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:36,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a27f26[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:36,971 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:36,971 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:36,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:36,973 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:36,973 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,973 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:36,975 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:36,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:36,977 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:36,977 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:36,977 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:36,977 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:36,977 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:36,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:36,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:36,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:36,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:36,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:36,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:36,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:36,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:36,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:36,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:36,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:36,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:36,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/5af046a9d7464f6c9ec53d61cca4c3a0 2023-04-01 03:56:37,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/6d7c5be4fdac43ddbe536e106d125f15 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15 2023-04-01 03:56:37,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:37,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:37,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:37,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15, entries=250, sequenceid=436, filesize=16.7 K 2023-04-01 03:56:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:37,096 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:37,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:37,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:37,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:37,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/6303cb0749b04a5787ee439b7e07dba1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1 2023-04-01 03:56:37,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1, entries=150, sequenceid=436, filesize=11.9 K 2023-04-01 03:56:37,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:37,102 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/5af046a9d7464f6c9ec53d61cca4c3a0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0 2023-04-01 03:56:37,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:37,103 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:37,103 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:37,103 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:37,104 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,104 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:37,104 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:37,104 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:37,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:37,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:37,107 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,107 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0, entries=150, sequenceid=436, filesize=11.9 K 2023-04-01 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:37,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:37,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411cf654[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:37,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:37,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d055bdf2a6c005123440ac61e55072e8 in 881ms, sequenceid=436, compaction requested=true 2023-04-01 03:56:37,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:37,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:37,108 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:37,109 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:37,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:37,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:37,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,110 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42290 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:37,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:37,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:37,110 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,110 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:37,110 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:37,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:37,110 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:37,110 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:37,110 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,110 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,110 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/f1949e0d6549428aa0bba1ee026be679, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.5 K 2023-04-01 03:56:37,110 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/04304e0606614882a81b119402e85191, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=41.3 K 2023-04-01 03:56:37,111 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f1949e0d6549428aa0bba1ee026be679, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:37,111 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 04304e0606614882a81b119402e85191, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:37,111 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 29aeb66d164148399da76864dc5c2b3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680321394776 2023-04-01 03:56:37,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:37,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:37,111 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:37,111 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 3b2f8d16bc68432f8777ba9435488854, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680321394776 2023-04-01 03:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:37,111 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6303cb0749b04a5787ee439b7e07dba1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:37,112 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6d7c5be4fdac43ddbe536e106d125f15, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:37,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:37,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:37,114 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:37,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:37,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:37,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:37,114 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:37,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:37,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:37,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:37,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,120 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:37,121 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#85 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:37,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:37,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:37,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:37,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:37,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:37,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:37,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:37,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:37,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,223 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:37,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:37,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:37,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:37,227 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:37,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:37,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:37,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:37,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,228 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:37,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:37,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:37,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:37,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:37,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:37,234 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,234 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:37,234 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:56:37,234 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:37,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:37,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:37,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:37,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:37,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:37,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:37,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321457422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321457423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321457525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321457526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,543 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/f306f9896ca446d9aa039b1f7ec359f4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f306f9896ca446d9aa039b1f7ec359f4 2023-04-01 03:56:37,550 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into f306f9896ca446d9aa039b1f7ec359f4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:37,550 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:37,550 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321397108; duration=0sec 2023-04-01 03:56:37,550 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:37,550 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:37,552 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/729be715c2f94994a9ae743c1ff46b13 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/729be715c2f94994a9ae743c1ff46b13 2023-04-01 03:56:37,552 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:37,552 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:37,552 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:37,552 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1c5366585b374ca0a30865df974f8283, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.5 K 2023-04-01 03:56:37,553 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1c5366585b374ca0a30865df974f8283, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680321394140 2023-04-01 03:56:37,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 946c04f879bc4bc994286fe799e5d724, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680321394776 2023-04-01 03:56:37,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5af046a9d7464f6c9ec53d61cca4c3a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:37,560 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 729be715c2f94994a9ae743c1ff46b13(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:37,560 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:37,560 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321397108; duration=0sec 2023-04-01 03:56:37,561 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:37,564 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:37,658 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/5f8800c9ac2d4862b85248af13827498 2023-04-01 03:56:37,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321457728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:37,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321457728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:37,975 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/f12d92cbe87745e588a6d956d29e71cc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f12d92cbe87745e588a6d956d29e71cc 2023-04-01 03:56:37,981 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into f12d92cbe87745e588a6d956d29e71cc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:37,981 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:37,981 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321397109; duration=0sec 2023-04-01 03:56:37,981 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:38,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321458031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321458032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,070 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/3866e0bb80854a64a0cb8e814a96cc12 2023-04-01 03:56:38,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43544 deadline: 1680321458429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,431 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:38,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43502 deadline: 1680321458456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,460 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:38,488 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/caa0303441fe4b449c99f3e425f80fe4 2023-04-01 03:56:38,493 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/5f8800c9ac2d4862b85248af13827498 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498 2023-04-01 03:56:38,497 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498, entries=150, sequenceid=454, filesize=11.9 K 2023-04-01 03:56:38,498 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/3866e0bb80854a64a0cb8e814a96cc12 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12 2023-04-01 03:56:38,507 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12, entries=150, sequenceid=454, filesize=11.9 K 2023-04-01 03:56:38,508 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/caa0303441fe4b449c99f3e425f80fe4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4 2023-04-01 03:56:38,514 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4, entries=150, sequenceid=454, filesize=11.9 K 2023-04-01 03:56:38,514 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d055bdf2a6c005123440ac61e55072e8 in 1280ms, sequenceid=454, compaction requested=false 2023-04-01 03:56:38,515 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:38,515 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:38,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:38,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:38,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:38,515 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,517 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:38,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34225b53[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:38,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:38,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:38,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:38,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:38,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:38,522 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:38,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:38,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:38,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:38,525 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:38,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:38,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:38,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:38,525 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:38,526 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,526 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:38,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,526 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:38,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:38,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:38,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:38,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:38,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:38,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:38,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:38,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,531 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:38,531 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:38,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:38,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:38,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:38,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:56:38,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:38,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:38,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:38,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e0a7dfdbc6d64302a6139e27cac89a13 2023-04-01 03:56:38,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/ff9799abc1074eed9366ce0d71c13fa3 2023-04-01 03:56:38,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321458570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321458570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/7094c734d40248279b2efcf6c2e1ef24 2023-04-01 03:56:38,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/e0a7dfdbc6d64302a6139e27cac89a13 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13 2023-04-01 03:56:38,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13, entries=200, sequenceid=473, filesize=14.3 K 2023-04-01 03:56:38,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/ff9799abc1074eed9366ce0d71c13fa3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3 2023-04-01 03:56:38,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3, entries=150, sequenceid=473, filesize=11.9 K 2023-04-01 03:56:38,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/7094c734d40248279b2efcf6c2e1ef24 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24 2023-04-01 03:56:38,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24, entries=150, sequenceid=473, filesize=11.9 K 2023-04-01 03:56:38,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for d055bdf2a6c005123440ac61e55072e8 in 86ms, sequenceid=473, compaction requested=true 2023-04-01 03:56:38,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:38,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:38,623 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:38,623 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:38,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:38,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:38,625 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:38,625 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:38,625 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:38,625 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:38,625 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:38,625 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:38,625 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f306f9896ca446d9aa039b1f7ec359f4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=39.0 K 2023-04-01 03:56:38,625 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/729be715c2f94994a9ae743c1ff46b13, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.6 K 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 729be715c2f94994a9ae743c1ff46b13, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f306f9896ca446d9aa039b1f7ec359f4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 3866e0bb80854a64a0cb8e814a96cc12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680321396250 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5f8800c9ac2d4862b85248af13827498, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680321396250 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ff9799abc1074eed9366ce0d71c13fa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397421 2023-04-01 03:56:38,626 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e0a7dfdbc6d64302a6139e27cac89a13, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397409 2023-04-01 03:56:38,631 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:38,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:38,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:38,636 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#93 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:38,646 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:38,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:38,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:38,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:38,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:38,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:38,657 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:38,658 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:38,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:38,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:38,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:38,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:38,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:38,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:38,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:38,670 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:38,670 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:38,670 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:38,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:38,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:38,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:38,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:38,675 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:56:38,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:38,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:38,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:38,696 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/ee87a9252ecc4ad18aa50d97b4718eeb 2023-04-01 03:56:38,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321458696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321458703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321458804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:38,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321458808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321459007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321459012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,077 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/e1d20b64e6094ea280e2703e5b2ed715 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/e1d20b64e6094ea280e2703e5b2ed715 2023-04-01 03:56:39,093 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into e1d20b64e6094ea280e2703e5b2ed715(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:39,093 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:39,093 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=13, startTime=1680321398623; duration=0sec 2023-04-01 03:56:39,093 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:39,094 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:56:39,094 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/80d23854ae364024a094aced95a50bf0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/80d23854ae364024a094aced95a50bf0 2023-04-01 03:56:39,094 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:56:39,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:39,095 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:39,095 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f12d92cbe87745e588a6d956d29e71cc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=36.6 K 2023-04-01 03:56:39,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f12d92cbe87745e588a6d956d29e71cc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680321395910 2023-04-01 03:56:39,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting caa0303441fe4b449c99f3e425f80fe4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680321396250 2023-04-01 03:56:39,096 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7094c734d40248279b2efcf6c2e1ef24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397421 2023-04-01 03:56:39,099 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 80d23854ae364024a094aced95a50bf0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:39,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:39,099 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=13, startTime=1680321398623; duration=0sec 2023-04-01 03:56:39,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:39,102 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:39,116 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/c7078efd2e4d4b9b950574b58377d6b4 2023-04-01 03:56:39,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321459311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321459314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,513 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/ff381de2e16749b08438d262694bcb5a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ff381de2e16749b08438d262694bcb5a 2023-04-01 03:56:39,518 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into ff381de2e16749b08438d262694bcb5a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:39,518 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:39,518 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=13, startTime=1680321398624; duration=0sec 2023-04-01 03:56:39,519 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:39,530 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/279de343cff9402da020e4e422472919 2023-04-01 03:56:39,535 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/ee87a9252ecc4ad18aa50d97b4718eeb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb 2023-04-01 03:56:39,541 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb, entries=150, sequenceid=492, filesize=11.9 K 2023-04-01 03:56:39,542 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/c7078efd2e4d4b9b950574b58377d6b4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4 2023-04-01 03:56:39,547 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4, entries=150, sequenceid=492, filesize=11.9 K 2023-04-01 03:56:39,548 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/279de343cff9402da020e4e422472919 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919 2023-04-01 03:56:39,552 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919, entries=150, sequenceid=492, filesize=11.9 K 2023-04-01 03:56:39,553 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d055bdf2a6c005123440ac61e55072e8 in 879ms, sequenceid=492, compaction requested=false 2023-04-01 03:56:39,553 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:39,553 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:39,553 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:39,553 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:39,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:39,555 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:39,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:39,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb6f230[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:39,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:39,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:39,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:39,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:39,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:39,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:39,560 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:39,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:39,562 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:39,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:39,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:39,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:39,562 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:39,564 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,564 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:39,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,564 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:39,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:39,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:39,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:39,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:39,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:39,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:39,568 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:39,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:39,568 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:39,568 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:39,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:39,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:39,668 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:39,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:39,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:39,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:39,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:39,671 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:39,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:39,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:39,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:39,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:39,672 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:39,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:39,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:39,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:39,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:39,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:39,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:39,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:39,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:39,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:39,673 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:39,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:39,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:39,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:39,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321459853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321459853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321459954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:39,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321459957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,081 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/4b01b1fe49d0406bb5224a55f36ea5fe 2023-04-01 03:56:40,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321460157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321460159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321460460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321460462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,494 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/a212c030dde04a62b4be80529d395d17 2023-04-01 03:56:40,511 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/4e152c0097c64735b8c821a316b5b399 2023-04-01 03:56:40,516 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/4b01b1fe49d0406bb5224a55f36ea5fe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe 2023-04-01 03:56:40,520 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe, entries=150, sequenceid=508, filesize=11.9 K 2023-04-01 03:56:40,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/a212c030dde04a62b4be80529d395d17 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17 2023-04-01 03:56:40,525 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17, entries=150, sequenceid=508, filesize=11.9 K 2023-04-01 03:56:40,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/4e152c0097c64735b8c821a316b5b399 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399 2023-04-01 03:56:40,531 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399, entries=150, sequenceid=508, filesize=11.9 K 2023-04-01 03:56:40,531 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d055bdf2a6c005123440ac61e55072e8 in 858ms, sequenceid=508, compaction requested=true 2023-04-01 03:56:40,531 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:40,531 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:40,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:40,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:40,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:40,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:40,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:40,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:40,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7fe9cf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:40,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:40,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:40,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:40,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:40,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:40,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:40,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:40,538 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:40,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:40,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:40,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:40,538 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:40,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,555 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:40,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:40,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:40,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:40,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:40,566 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:40,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:40,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,566 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:40,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:40,666 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:40,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:40,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:40,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:40,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:40,668 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:40,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:40,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:40,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:40,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:40,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:40,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:40,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:40,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:40,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:40,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:40,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:40,672 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:40,673 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:40,673 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:40,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:40,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:40,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:40,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321460982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:40,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:40,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321460985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,084 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/48d9412bb42e4fc5acb0e3b1611770ee 2023-04-01 03:56:41,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321461086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321461087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,095 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/84ac71b024704234b3e235ad99fa6a45 2023-04-01 03:56:41,107 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2529d1fc1b124f4c929149945706d953 2023-04-01 03:56:41,111 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/48d9412bb42e4fc5acb0e3b1611770ee as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee 2023-04-01 03:56:41,115 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee, entries=150, sequenceid=527, filesize=11.9 K 2023-04-01 03:56:41,116 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/84ac71b024704234b3e235ad99fa6a45 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45 2023-04-01 03:56:41,120 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45, entries=150, sequenceid=527, filesize=11.9 K 2023-04-01 03:56:41,121 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/2529d1fc1b124f4c929149945706d953 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953 2023-04-01 03:56:41,125 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953, entries=150, sequenceid=527, filesize=11.9 K 2023-04-01 03:56:41,126 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d055bdf2a6c005123440ac61e55072e8 in 453ms, sequenceid=527, compaction requested=true 2023-04-01 03:56:41,126 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:41,126 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:41,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:41,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:41,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:41,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:41,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,129 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:41,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48770c02[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:41,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,132 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:41,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:41,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:41,132 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,132 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:41,134 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:41,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:41,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:41,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:41,137 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:41,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:41,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:41,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:41,137 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:41,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:41,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:41,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:41,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:41,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:41,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:41,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:41,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:41,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,144 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,144 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:41,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:56:41,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:41,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:41,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:41,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:41,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:41,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:41,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321461198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:41,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:41,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:41,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:41,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:41,246 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:41,246 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:41,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:41,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:41,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,247 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,250 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:41,250 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:41,250 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,250 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:41,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:41,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:41,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:41,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:41,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:41,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:41,251 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:41,251 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:41,252 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. started... 2023-04-01 03:56:41,252 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. as already flushing 2023-04-01 03:56:41,252 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:41,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:41,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:41,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:41,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:41,253 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,253 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,253 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,253 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:41,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:41,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:41,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e293e9c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:41,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,255 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:41,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:41,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:41,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:41,256 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:41,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:41,258 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:41,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:41,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:41,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:41,258 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:41,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:41,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:41,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:41,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,261 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:41,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:41,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:41,263 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:41,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:41,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:41,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:41,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:41,274 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26c4dd91 to 127.0.0.1:50457 2023-04-01 03:56:41,275 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:41,275 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a15fbcc to 127.0.0.1:50457 2023-04-01 03:56:41,275 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:41,276 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40d2a406 to 127.0.0.1:50457 2023-04-01 03:56:41,276 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:41,279 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c201e2 to 127.0.0.1:50457 2023-04-01 03:56:41,279 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:41,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321461288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321461291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321461301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:41,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:41,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321461503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/ebdf847709574dcba94db1e534959b39 2023-04-01 03:56:41,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321461590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321461592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321461808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:41,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/8296c6e50e33455dbf2752baddfddf26 2023-04-01 03:56:42,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43514 deadline: 1680321462094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:42,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43566 deadline: 1680321462095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:42,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:42,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43484 deadline: 1680321462314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:42,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/ae66e9e376264ef5b220e201931db848 2023-04-01 03:56:42,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/ebdf847709574dcba94db1e534959b39 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39 2023-04-01 03:56:42,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39, entries=200, sequenceid=541, filesize=14.3 K 2023-04-01 03:56:42,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/8296c6e50e33455dbf2752baddfddf26 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26 2023-04-01 03:56:42,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26, entries=150, sequenceid=541, filesize=11.9 K 2023-04-01 03:56:42,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/ae66e9e376264ef5b220e201931db848 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848 2023-04-01 03:56:42,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848, entries=150, sequenceid=541, filesize=11.9 K 2023-04-01 03:56:42,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d055bdf2a6c005123440ac61e55072e8 in 1293ms, sequenceid=541, compaction requested=true 2023-04-01 03:56:42,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:42,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:42,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:42,443 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:56:42,443 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:56:42,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:42,445 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64426 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:56:42,445 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:56:42,445 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/A is initiating minor compaction (all files) 2023-04-01 03:56:42,445 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/B is initiating minor compaction (all files) 2023-04-01 03:56:42,445 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/A in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:42,445 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/B in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:42,446 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/80d23854ae364024a094aced95a50bf0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=62.9 K 2023-04-01 03:56:42,446 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/e1d20b64e6094ea280e2703e5b2ed715, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=60.5 K 2023-04-01 03:56:42,446 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 80d23854ae364024a094aced95a50bf0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397421 2023-04-01 03:56:42,446 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e1d20b64e6094ea280e2703e5b2ed715, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397421 2023-04-01 03:56:42,447 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ee87a9252ecc4ad18aa50d97b4718eeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680321398562 2023-04-01 03:56:42,447 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c7078efd2e4d4b9b950574b58377d6b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680321398562 2023-04-01 03:56:42,447 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4b01b1fe49d0406bb5224a55f36ea5fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680321398682 2023-04-01 03:56:42,447 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a212c030dde04a62b4be80529d395d17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680321398682 2023-04-01 03:56:42,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 48d9412bb42e4fc5acb0e3b1611770ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680321399851 2023-04-01 03:56:42,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 84ac71b024704234b3e235ad99fa6a45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680321399851 2023-04-01 03:56:42,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ebdf847709574dcba94db1e534959b39, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680321400969 2023-04-01 03:56:42,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8296c6e50e33455dbf2752baddfddf26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680321400969 2023-04-01 03:56:42,462 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#A#compaction#108 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:42,462 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#B#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:42,882 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/55465a5c65a34a12b0c01aa0d96cf2aa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/55465a5c65a34a12b0c01aa0d96cf2aa 2023-04-01 03:56:42,883 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/36fe9945d9834ce995a50918e512de51 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/36fe9945d9834ce995a50918e512de51 2023-04-01 03:56:42,888 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/B of d055bdf2a6c005123440ac61e55072e8 into 55465a5c65a34a12b0c01aa0d96cf2aa(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:42,888 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/A of d055bdf2a6c005123440ac61e55072e8 into 36fe9945d9834ce995a50918e512de51(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:42,889 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:42,889 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:42,889 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/B, priority=11, startTime=1680321402443; duration=0sec 2023-04-01 03:56:42,889 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/A, priority=11, startTime=1680321402443; duration=0sec 2023-04-01 03:56:42,889 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:42,889 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:42,889 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:56:42,891 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:56:42,891 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): d055bdf2a6c005123440ac61e55072e8/C is initiating minor compaction (all files) 2023-04-01 03:56:42,891 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d055bdf2a6c005123440ac61e55072e8/C in TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:42,891 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ff381de2e16749b08438d262694bcb5a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp, totalSize=60.5 K 2023-04-01 03:56:42,892 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ff381de2e16749b08438d262694bcb5a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680321397421 2023-04-01 03:56:42,892 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 279de343cff9402da020e4e422472919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1680321398562 2023-04-01 03:56:42,893 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4e152c0097c64735b8c821a316b5b399, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680321398682 2023-04-01 03:56:42,893 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2529d1fc1b124f4c929149945706d953, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1680321399851 2023-04-01 03:56:42,893 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ae66e9e376264ef5b220e201931db848, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680321400969 2023-04-01 03:56:42,906 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): d055bdf2a6c005123440ac61e55072e8#C#compaction#110 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:43,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:43,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:56:43,101 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6931edc7 to 127.0.0.1:50457 2023-04-01 03:56:43,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:43,102 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:43,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:43,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:43,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:43,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:43,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:43,105 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51210e37 to 127.0.0.1:50457 2023-04-01 03:56:43,105 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:43,327 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32ee6250 to 127.0.0.1:50457 2023-04-01 03:56:43,327 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:43,327 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/b33c2a3fd0d84610a70f93620c8d2fd1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/b33c2a3fd0d84610a70f93620c8d2fd1 2023-04-01 03:56:43,335 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d055bdf2a6c005123440ac61e55072e8/C of d055bdf2a6c005123440ac61e55072e8 into b33c2a3fd0d84610a70f93620c8d2fd1(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:43,336 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:43,336 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8., storeName=d055bdf2a6c005123440ac61e55072e8/C, priority=11, startTime=1680321402443; duration=0sec 2023-04-01 03:56:43,336 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:43,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/476ecb633eeb4cbd85b1410f2a7632f2 2023-04-01 03:56:43,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/88404a85a05b4b0c939367d2a394059e 2023-04-01 03:56:44,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/00537e0fbb344fd889802badd4ff79e4 2023-04-01 03:56:44,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/476ecb633eeb4cbd85b1410f2a7632f2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/476ecb633eeb4cbd85b1410f2a7632f2 2023-04-01 03:56:44,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/476ecb633eeb4cbd85b1410f2a7632f2, entries=150, sequenceid=562, filesize=11.9 K 2023-04-01 03:56:44,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/88404a85a05b4b0c939367d2a394059e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/88404a85a05b4b0c939367d2a394059e 2023-04-01 03:56:44,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/88404a85a05b4b0c939367d2a394059e, entries=150, sequenceid=562, filesize=11.9 K 2023-04-01 03:56:44,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/00537e0fbb344fd889802badd4ff79e4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/00537e0fbb344fd889802badd4ff79e4 2023-04-01 03:56:44,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/00537e0fbb344fd889802badd4ff79e4, entries=150, sequenceid=562, filesize=11.9 K 2023-04-01 03:56:44,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for d055bdf2a6c005123440ac61e55072e8 in 1279ms, sequenceid=562, compaction requested=false 2023-04-01 03:56:44,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:48,521 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f7664e5 to 127.0.0.1:50457 2023-04-01 03:56:48,521 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:48,561 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0faf312d to 127.0.0.1:50457 2023-04-01 03:56:48,561 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5334 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5159 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 2252 2023-04-01 03:56:48,561 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 6752 rows 2023-04-01 03:56:48,562 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 2240 2023-04-01 03:56:48,562 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 6720 rows 2023-04-01 03:56:48,562 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:56:48,562 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a33cf3a to 127.0.0.1:50457 2023-04-01 03:56:48,562 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:56:48,565 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:56:48,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:56:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:48,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321408579"}]},"ts":"1680321408579"} 2023-04-01 03:56:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:48,580 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:56:48,582 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:56:48,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, UNASSIGN}] 2023-04-01 03:56:48,585 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, UNASSIGN 2023-04-01 03:56:48,586 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d055bdf2a6c005123440ac61e55072e8, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:48,586 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321408586"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321408586"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321408586"}]},"ts":"1680321408586"} 2023-04-01 03:56:48,588 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:48,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:48,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:48,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing d055bdf2a6c005123440ac61e55072e8, disabling compactions & flushes 2023-04-01 03:56:48,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:48,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:48,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. after waiting 0 ms 2023-04-01 03:56:48,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:48,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing d055bdf2a6c005123440ac61e55072e8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 03:56:48,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=A 2023-04-01 03:56:48,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:48,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=B 2023-04-01 03:56:48,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:48,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d055bdf2a6c005123440ac61e55072e8, store=C 2023-04-01 03:56:48,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:48,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:49,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/94f79963ba064209b7c79dc89942b173 2023-04-01 03:56:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:49,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/80ad12fb6e2049abbdf68b654ecf8621 2023-04-01 03:56:49,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:50,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/6b5cd0be0b594fa9a5b1075020f8dedf 2023-04-01 03:56:50,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/A/94f79963ba064209b7c79dc89942b173 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/94f79963ba064209b7c79dc89942b173 2023-04-01 03:56:50,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/94f79963ba064209b7c79dc89942b173, entries=150, sequenceid=570, filesize=11.9 K 2023-04-01 03:56:50,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/B/80ad12fb6e2049abbdf68b654ecf8621 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/80ad12fb6e2049abbdf68b654ecf8621 2023-04-01 03:56:50,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/80ad12fb6e2049abbdf68b654ecf8621, entries=150, sequenceid=570, filesize=11.9 K 2023-04-01 03:56:50,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/.tmp/C/6b5cd0be0b594fa9a5b1075020f8dedf as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6b5cd0be0b594fa9a5b1075020f8dedf 2023-04-01 03:56:50,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6b5cd0be0b594fa9a5b1075020f8dedf, entries=150, sequenceid=570, filesize=11.9 K 2023-04-01 03:56:50,043 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d055bdf2a6c005123440ac61e55072e8 in 1292ms, sequenceid=570, compaction requested=true 2023-04-01 03:56:50,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/964f60818e044232b7c1bf4b00d06bb2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8a2bb1f1f217415cbac30681325c4d02, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/a0e83d4a981143a19123d4495538f32d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5580839f0533471da6cde5db00d2ea0a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7f1faed54b804db8be9e7c3121ab1db3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/0b5f1cc8fd804faebfa1b26e43469946, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/04304e0606614882a81b119402e85191, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f306f9896ca446d9aa039b1f7ec359f4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/80d23854ae364024a094aced95a50bf0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39] to archive 2023-04-01 03:56:50,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:56:50,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0274ed747ae468989a2be5410259b51 2023-04-01 03:56:50,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/965ddf2678d34eb0897ddc1440e68895 2023-04-01 03:56:50,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c7a40749a83140d89e40d2b32d2d820d 2023-04-01 03:56:50,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/964f60818e044232b7c1bf4b00d06bb2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/964f60818e044232b7c1bf4b00d06bb2 2023-04-01 03:56:50,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/febdb947f7e140b0bedb0a9aaa842179 2023-04-01 03:56:50,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8a2bb1f1f217415cbac30681325c4d02 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8a2bb1f1f217415cbac30681325c4d02 2023-04-01 03:56:50,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/42ed8096c4d743bfbd422d00b261b3d1 2023-04-01 03:56:50,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dce98ddcf3b747dcbd6e699733e62de8 2023-04-01 03:56:50,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/a0e83d4a981143a19123d4495538f32d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/a0e83d4a981143a19123d4495538f32d 2023-04-01 03:56:50,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5457617001f64faca57df208a40adb66 2023-04-01 03:56:50,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/dd5c7a78d32340d0b13884d0ebc440d7 2023-04-01 03:56:50,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7e414969940a443c937c2451d37a89a8 2023-04-01 03:56:50,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5580839f0533471da6cde5db00d2ea0a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5580839f0533471da6cde5db00d2ea0a 2023-04-01 03:56:50,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/79a3b5c5a5d643e4b3f58b96c077be40 2023-04-01 03:56:50,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4dc8e54b3e3e4173a1f5ff79d57b767c 2023-04-01 03:56:50,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/c854e8adb91a4db8927ef3391db3c4b0 2023-04-01 03:56:50,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7f1faed54b804db8be9e7c3121ab1db3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/7f1faed54b804db8be9e7c3121ab1db3 2023-04-01 03:56:50,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e65a491314b9473a990729bfff2d4381 2023-04-01 03:56:50,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/56a7eb07aaa840e792bea91521ca4bec 2023-04-01 03:56:50,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/0b5f1cc8fd804faebfa1b26e43469946 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/0b5f1cc8fd804faebfa1b26e43469946 2023-04-01 03:56:50,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/b3d54ddcb07442619760bb457f44a92e 2023-04-01 03:56:50,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/8016c39f00e54d689a604884dba4db87 2023-04-01 03:56:50,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/cecc23cc24ea42b7b63a756c2c488ad8 2023-04-01 03:56:50,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/534547d616f1466b911caad61959a4ff 2023-04-01 03:56:50,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f7f44fc69f264abca62512f47fa24410 2023-04-01 03:56:50,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/04304e0606614882a81b119402e85191 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/04304e0606614882a81b119402e85191 2023-04-01 03:56:50,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/3b2f8d16bc68432f8777ba9435488854 2023-04-01 03:56:50,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/6d7c5be4fdac43ddbe536e106d125f15 2023-04-01 03:56:50,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f306f9896ca446d9aa039b1f7ec359f4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/f306f9896ca446d9aa039b1f7ec359f4 2023-04-01 03:56:50,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/5f8800c9ac2d4862b85248af13827498 2023-04-01 03:56:50,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/e0a7dfdbc6d64302a6139e27cac89a13 2023-04-01 03:56:50,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/80d23854ae364024a094aced95a50bf0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/80d23854ae364024a094aced95a50bf0 2023-04-01 03:56:50,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ee87a9252ecc4ad18aa50d97b4718eeb 2023-04-01 03:56:50,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/4b01b1fe49d0406bb5224a55f36ea5fe 2023-04-01 03:56:50,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/48d9412bb42e4fc5acb0e3b1611770ee 2023-04-01 03:56:50,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/ebdf847709574dcba94db1e534959b39 2023-04-01 03:56:50,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c5677728ea984a73ab6af7136172897e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/39adf773958840818c2e2c8f780bd1d4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1b86764df117470691621fd17518a4b3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1eefbc00d9564bb79e4f312bcaf2784f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/94341de8cc654de8a146f051a7eb4a75, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/9a405389d3d348588e379691fb773c88, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/f1949e0d6549428aa0bba1ee026be679, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/729be715c2f94994a9ae743c1ff46b13, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/e1d20b64e6094ea280e2703e5b2ed715, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26] to archive 2023-04-01 03:56:50,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:56:50,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/383944d33c4c4ed68bd77fac16231e35 2023-04-01 03:56:50,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/71d0730e2e0b41459fc2bd05779ab9ac 2023-04-01 03:56:50,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c5677728ea984a73ab6af7136172897e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c5677728ea984a73ab6af7136172897e 2023-04-01 03:56:50,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6449d9639b8a418e8037bd82c5004661 2023-04-01 03:56:50,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/31c3aca2c49242e3a11f4bd7c2d38ce8 2023-04-01 03:56:50,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/39adf773958840818c2e2c8f780bd1d4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/39adf773958840818c2e2c8f780bd1d4 2023-04-01 03:56:50,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/81d0f3977fad40798f73adbf2762ed93 2023-04-01 03:56:50,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/fe8fda2cbe6b49fa8fba85bffa58edc1 2023-04-01 03:56:50,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1b86764df117470691621fd17518a4b3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1b86764df117470691621fd17518a4b3 2023-04-01 03:56:50,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/97c9836351cf4734ae2330b61ee57b65 2023-04-01 03:56:50,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b9875610403b47ae871849244fc3112d 2023-04-01 03:56:50,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1eefbc00d9564bb79e4f312bcaf2784f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/1eefbc00d9564bb79e4f312bcaf2784f 2023-04-01 03:56:50,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/0386015aaa364128bb6d4930717f85e1 2023-04-01 03:56:50,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/52b91a7352364655a56b906d21348c5f 2023-04-01 03:56:50,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8aaffd4f2d19438b980e041642060117 2023-04-01 03:56:50,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/94341de8cc654de8a146f051a7eb4a75 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/94341de8cc654de8a146f051a7eb4a75 2023-04-01 03:56:50,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/67cbd2bc04494d23bdb2177865a7a2c4 2023-04-01 03:56:50,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/4cd69964dbb942edb0f0e7b42baf1fd4 2023-04-01 03:56:50,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/9a405389d3d348588e379691fb773c88 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/9a405389d3d348588e379691fb773c88 2023-04-01 03:56:50,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/926ba8e3356a4b27bad55769a6e32909 2023-04-01 03:56:50,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b5404f09bd0d4375971f8d4c4b508930 2023-04-01 03:56:50,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/48fa4119f47146f5bc9894151efa7406 2023-04-01 03:56:50,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/101c788d681b43e8a4cef5807c5384fa 2023-04-01 03:56:50,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/b385cf79096a442cadee5deb781c8a51 2023-04-01 03:56:50,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/f1949e0d6549428aa0bba1ee026be679 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/f1949e0d6549428aa0bba1ee026be679 2023-04-01 03:56:50,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/25c3c933fdad4df898042c0c4139bd65 2023-04-01 03:56:50,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/29aeb66d164148399da76864dc5c2b3f 2023-04-01 03:56:50,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/729be715c2f94994a9ae743c1ff46b13 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/729be715c2f94994a9ae743c1ff46b13 2023-04-01 03:56:50,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/6303cb0749b04a5787ee439b7e07dba1 2023-04-01 03:56:50,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/3866e0bb80854a64a0cb8e814a96cc12 2023-04-01 03:56:50,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/e1d20b64e6094ea280e2703e5b2ed715 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/e1d20b64e6094ea280e2703e5b2ed715 2023-04-01 03:56:50,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/ff9799abc1074eed9366ce0d71c13fa3 2023-04-01 03:56:50,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/c7078efd2e4d4b9b950574b58377d6b4 2023-04-01 03:56:50,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/a212c030dde04a62b4be80529d395d17 2023-04-01 03:56:50,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/84ac71b024704234b3e235ad99fa6a45 2023-04-01 03:56:50,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/8296c6e50e33455dbf2752baddfddf26 2023-04-01 03:56:50,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e975b45d61354b17ac39ae3c93618e9e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f78469fc1f184653b3dabb163fdc049a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/143519f9415a4eb0b8817d498372b527, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6697c32ecb30449c9305c8d9b0442b49, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/75c5ecb2545f4188a4775a6dfb3b94a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e609ca41bc0e4c78bde1ac291bb0d27c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1c5366585b374ca0a30865df974f8283, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f12d92cbe87745e588a6d956d29e71cc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ff381de2e16749b08438d262694bcb5a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848] to archive 2023-04-01 03:56:50,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:56:50,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1a42a4f61cec4ab6a6d6aa02ac6e13dc 2023-04-01 03:56:50,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/71ee50bf32c3483bbd756d31572c7b21 2023-04-01 03:56:50,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e975b45d61354b17ac39ae3c93618e9e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e975b45d61354b17ac39ae3c93618e9e 2023-04-01 03:56:50,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/037506c5ee914341a6fad2057d8af12f 2023-04-01 03:56:50,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/13dc9e7bc380458ba83b43d3c5a8e215 2023-04-01 03:56:50,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f78469fc1f184653b3dabb163fdc049a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f78469fc1f184653b3dabb163fdc049a 2023-04-01 03:56:50,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/a66b2fcd7b1341b6991b5bbe1acfd22d 2023-04-01 03:56:50,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2238ffc2c9f2481198e78dd300baefc9 2023-04-01 03:56:50,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5ff18ba99ede4814ba8c5523d76de39e 2023-04-01 03:56:50,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/143519f9415a4eb0b8817d498372b527 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/143519f9415a4eb0b8817d498372b527 2023-04-01 03:56:50,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/0970e700074744c08ae05d2c0b32d3ce 2023-04-01 03:56:50,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/c936130a200b43118ad2c6e69993bffe 2023-04-01 03:56:50,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6697c32ecb30449c9305c8d9b0442b49 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6697c32ecb30449c9305c8d9b0442b49 2023-04-01 03:56:50,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e2ea74deb9e1494b93a735ab1d7860b8 2023-04-01 03:56:50,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/8a2ce1435e774fd2ba3e2baa73b48daa 2023-04-01 03:56:50,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/75c5ecb2545f4188a4775a6dfb3b94a5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/75c5ecb2545f4188a4775a6dfb3b94a5 2023-04-01 03:56:50,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/441644d65868475ea746645bb0cd9fe3 2023-04-01 03:56:50,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2d5b6d62ba9747b9bfd0ab52eaeae5f8 2023-04-01 03:56:50,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e609ca41bc0e4c78bde1ac291bb0d27c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/e609ca41bc0e4c78bde1ac291bb0d27c 2023-04-01 03:56:50,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/91a3b3077ad74627bcad7306f99507ff 2023-04-01 03:56:50,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae177a8de0e7451884c90dc1136911bc 2023-04-01 03:56:50,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/08a2f9bfc18e4252a6c1214e55299926 2023-04-01 03:56:50,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/bb22d892bc6e4134bb60a06a9edafefe 2023-04-01 03:56:50,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/89ef8a4cdedc451a8c0d1d8e023684bf 2023-04-01 03:56:50,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1c5366585b374ca0a30865df974f8283 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/1c5366585b374ca0a30865df974f8283 2023-04-01 03:56:50,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/db8e103a4a284b1797aa285ac330d02d 2023-04-01 03:56:50,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/946c04f879bc4bc994286fe799e5d724 2023-04-01 03:56:50,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f12d92cbe87745e588a6d956d29e71cc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/f12d92cbe87745e588a6d956d29e71cc 2023-04-01 03:56:50,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/5af046a9d7464f6c9ec53d61cca4c3a0 2023-04-01 03:56:50,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/caa0303441fe4b449c99f3e425f80fe4 2023-04-01 03:56:50,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ff381de2e16749b08438d262694bcb5a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ff381de2e16749b08438d262694bcb5a 2023-04-01 03:56:50,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/7094c734d40248279b2efcf6c2e1ef24 2023-04-01 03:56:50,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/279de343cff9402da020e4e422472919 2023-04-01 03:56:50,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/4e152c0097c64735b8c821a316b5b399 2023-04-01 03:56:50,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/2529d1fc1b124f4c929149945706d953 2023-04-01 03:56:50,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/ae66e9e376264ef5b220e201931db848 2023-04-01 03:56:50,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/recovered.edits/573.seqid, newMaxSeqId=573, maxSeqId=1 2023-04-01 03:56:50,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8. 2023-04-01 03:56:50,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for d055bdf2a6c005123440ac61e55072e8: 2023-04-01 03:56:50,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:50,206 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=d055bdf2a6c005123440ac61e55072e8, regionState=CLOSED 2023-04-01 03:56:50,206 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321410206"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321410206"}]},"ts":"1680321410206"} 2023-04-01 03:56:50,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-01 03:56:50,209 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure d055bdf2a6c005123440ac61e55072e8, server=jenkins-hbase20.apache.org,36251,1680321375816 in 1.6200 sec 2023-04-01 03:56:50,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-01 03:56:50,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d055bdf2a6c005123440ac61e55072e8, UNASSIGN in 1.6250 sec 2023-04-01 03:56:50,211 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321410211"}]},"ts":"1680321410211"} 2023-04-01 03:56:50,212 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:56:50,215 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:56:50,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6440 sec 2023-04-01 03:56:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 03:56:50,689 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-01 03:56:50,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:56:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,705 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,706 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-01 03:56:50,712 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:50,716 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/recovered.edits] 2023-04-01 03:56:50,720 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/36fe9945d9834ce995a50918e512de51 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/36fe9945d9834ce995a50918e512de51 2023-04-01 03:56:50,721 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/476ecb633eeb4cbd85b1410f2a7632f2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/476ecb633eeb4cbd85b1410f2a7632f2 2023-04-01 03:56:50,723 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/94f79963ba064209b7c79dc89942b173 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/A/94f79963ba064209b7c79dc89942b173 2023-04-01 03:56:50,727 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/55465a5c65a34a12b0c01aa0d96cf2aa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/55465a5c65a34a12b0c01aa0d96cf2aa 2023-04-01 03:56:50,728 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/80ad12fb6e2049abbdf68b654ecf8621 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/80ad12fb6e2049abbdf68b654ecf8621 2023-04-01 03:56:50,730 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/88404a85a05b4b0c939367d2a394059e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/B/88404a85a05b4b0c939367d2a394059e 2023-04-01 03:56:50,733 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/00537e0fbb344fd889802badd4ff79e4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/00537e0fbb344fd889802badd4ff79e4 2023-04-01 03:56:50,735 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6b5cd0be0b594fa9a5b1075020f8dedf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/6b5cd0be0b594fa9a5b1075020f8dedf 2023-04-01 03:56:50,737 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/b33c2a3fd0d84610a70f93620c8d2fd1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/C/b33c2a3fd0d84610a70f93620c8d2fd1 2023-04-01 03:56:50,742 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/recovered.edits/573.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8/recovered.edits/573.seqid 2023-04-01 03:56:50,743 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/d055bdf2a6c005123440ac61e55072e8 2023-04-01 03:56:50,743 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:56:50,747 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,776 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:56:50,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-01 03:56:50,812 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:56:50,814 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,814 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:56:50,814 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321410814"}]},"ts":"9223372036854775807"} 2023-04-01 03:56:50,816 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:56:50,817 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d055bdf2a6c005123440ac61e55072e8, NAME => 'TestAcidGuarantees,,1680321379957.d055bdf2a6c005123440ac61e55072e8.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:56:50,817 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:56:50,817 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321410817"}]},"ts":"9223372036854775807"} 2023-04-01 03:56:50,819 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:56:50,822 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:50,823 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 122 msec 2023-04-01 03:56:51,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-01 03:56:51,012 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-01 03:56:51,033 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=229 (was 204) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a6fa178-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:36251-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2142721661_15 at /127.0.0.1:39740 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a6fa178-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a6fa178-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a6fa178-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=265 (was 197) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=7999 (was 8461) 2023-04-01 03:56:51,046 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=265, ProcessCount=167, AvailableMemoryMB=7999 2023-04-01 03:56:51,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:56:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:51,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:51,052 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:56:51,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-01 03:56:51,053 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:56:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:51,055 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:51,055 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 empty. 2023-04-01 03:56:51,056 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:51,057 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:56:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:51,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:51,473 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:51,478 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 57b058ac96577c51eb729e1451adecd4, NAME => 'TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:56:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:51,890 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:51,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 57b058ac96577c51eb729e1451adecd4, disabling compactions & flushes 2023-04-01 03:56:51,891 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:51,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:51,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. after waiting 0 ms 2023-04-01 03:56:51,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:51,891 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:51,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:51,899 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:56:51,901 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321411901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321411901"}]},"ts":"1680321411901"} 2023-04-01 03:56:51,904 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:56:51,905 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:56:51,906 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321411905"}]},"ts":"1680321411905"} 2023-04-01 03:56:51,907 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:56:51,913 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, ASSIGN}] 2023-04-01 03:56:51,915 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, ASSIGN 2023-04-01 03:56:51,916 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:56:52,068 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:52,069 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321412068"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321412068"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321412068"}]},"ts":"1680321412068"} 2023-04-01 03:56:52,073 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:52,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:52,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 57b058ac96577c51eb729e1451adecd4, NAME => 'TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:52,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:52,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,240 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,242 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A 2023-04-01 03:56:52,242 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A 2023-04-01 03:56:52,242 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:52,243 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName A 2023-04-01 03:56:52,243 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:52,243 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,245 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B 2023-04-01 03:56:52,245 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B 2023-04-01 03:56:52,246 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:52,246 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName B 2023-04-01 03:56:52,246 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:52,247 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,248 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C 2023-04-01 03:56:52,249 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C 2023-04-01 03:56:52,249 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:52,249 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName C 2023-04-01 03:56:52,250 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:52,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:52,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:56:52,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:52,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:56:52,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 57b058ac96577c51eb729e1451adecd4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73004549, jitterRate=0.08785255253314972}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:56:52,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:52,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., pid=18, masterSystemTime=1680321412226 2023-04-01 03:56:52,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:52,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:52,262 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:52,262 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321412262"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321412262"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321412262"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321412262"}]},"ts":"1680321412262"} 2023-04-01 03:56:52,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-01 03:56:52,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 in 191 msec 2023-04-01 03:56:52,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-01 03:56:52,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, ASSIGN in 353 msec 2023-04-01 03:56:52,270 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:56:52,270 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321412270"}]},"ts":"1680321412270"} 2023-04-01 03:56:52,272 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:56:52,274 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:56:52,275 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2250 sec 2023-04-01 03:56:53,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 03:56:53,165 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-01 03:56:53,167 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6902ce3c to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:53,179 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1409fd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:53,183 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:53,189 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44172, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:53,192 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:56:53,195 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:56:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:56:53,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:56:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 03:56:53,627 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:56:53,629 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:56:53,629 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:56:53,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 03:56:53,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, REOPEN/MOVE}] 2023-04-01 03:56:53,649 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, REOPEN/MOVE 2023-04-01 03:56:53,650 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:53,650 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321413650"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321413650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321413650"}]},"ts":"1680321413650"} 2023-04-01 03:56:53,651 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:53,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:53,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 57b058ac96577c51eb729e1451adecd4, disabling compactions & flushes 2023-04-01 03:56:53,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:53,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:53,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. after waiting 0 ms 2023-04-01 03:56:53,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:53,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 03:56:53,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:53,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:53,816 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 57b058ac96577c51eb729e1451adecd4 to self. 2023-04-01 03:56:53,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:53,818 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=CLOSED 2023-04-01 03:56:53,819 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321413818"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321413818"}]},"ts":"1680321413818"} 2023-04-01 03:56:53,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-01 03:56:53,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 in 169 msec 2023-04-01 03:56:53,824 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=true 2023-04-01 03:56:53,974 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:53,975 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321413974"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321413974"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321413974"}]},"ts":"1680321413974"} 2023-04-01 03:56:53,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:56:54,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 57b058ac96577c51eb729e1451adecd4, NAME => 'TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:56:54,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:56:54,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,148 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,149 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A 2023-04-01 03:56:54,149 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A 2023-04-01 03:56:54,149 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:54,152 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName A 2023-04-01 03:56:54,154 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:54,155 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,157 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B 2023-04-01 03:56:54,157 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B 2023-04-01 03:56:54,157 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:54,157 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName B 2023-04-01 03:56:54,158 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:54,158 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,160 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C 2023-04-01 03:56:54,160 DEBUG [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C 2023-04-01 03:56:54,160 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:56:54,160 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 57b058ac96577c51eb729e1451adecd4 columnFamilyName C 2023-04-01 03:56:54,161 INFO [StoreOpener-57b058ac96577c51eb729e1451adecd4-1] regionserver.HStore(310): Store=57b058ac96577c51eb729e1451adecd4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:56:54,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:56:54,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 57b058ac96577c51eb729e1451adecd4; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68219057, jitterRate=0.01654316484928131}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:56:54,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:54,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., pid=23, masterSystemTime=1680321414133 2023-04-01 03:56:54,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,171 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,171 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321414171"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321414171"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321414171"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321414171"}]},"ts":"1680321414171"} 2023-04-01 03:56:54,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-01 03:56:54,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 in 193 msec 2023-04-01 03:56:54,175 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-01 03:56:54,175 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, REOPEN/MOVE in 525 msec 2023-04-01 03:56:54,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-01 03:56:54,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 540 msec 2023-04-01 03:56:54,179 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 972 msec 2023-04-01 03:56:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-01 03:56:54,187 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x54ec4d71 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,192 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ca97129, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,193 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x63f0ce11 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,209 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61f8fa31, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,209 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6ec2774b to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,217 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b27bcc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,218 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3e0f4d47 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,225 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8f807de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,226 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x710671dd to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,254 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35a77fcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,254 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3659fc7a to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,274 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@380eb479, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,274 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x64b70ca7 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,292 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d749667, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,293 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x4fde0bbc to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,300 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13d0f4d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,301 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x58ee9cda to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:56:54,309 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6752ef90, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:56:54,316 DEBUG [hconnection-0x37165897-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,316 DEBUG [hconnection-0x41197484-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,316 DEBUG [hconnection-0x5a6855f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,317 DEBUG [hconnection-0x478f773c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,318 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,319 DEBUG [hconnection-0x1839d49a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,319 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,324 DEBUG [hconnection-0x5776d3f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,328 DEBUG [hconnection-0x6e2ca5b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,328 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,328 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,328 DEBUG [hconnection-0x21fce1f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,329 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,328 DEBUG [hconnection-0x32e0e024-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:56:54,329 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,331 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,333 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,335 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:56:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:54,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:54,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:54,367 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:54,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:54,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:54,369 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,369 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:54,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:54,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:54,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:56:54,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:54,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:54,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:54,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:54,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:54,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:54,383 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:54,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:56:54,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:54,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:54,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:54,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:54,384 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:54,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:56:54,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:54,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:56:54,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,400 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c5761b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:54,403 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:54,403 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,404 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:54,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:54,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:54,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:54,405 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:54,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:54,408 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:54,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:54,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:54,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:54,408 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:54,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321474407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321474409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321474409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321474410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321474411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:54,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:54,429 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,429 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:54,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,429 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:54,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321474513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321474514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321474514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321474515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321474514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,529 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:54,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:54,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:54,538 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:54,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:54,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:54,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:54,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:54,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:54,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:54,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:54,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:54,554 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,554 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:54,554 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:54,554 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:54,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:54,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e5d2fd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:54,577 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:54,577 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,590 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:54,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:54,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:54,590 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,590 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:54,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:54,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:54,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:54,595 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:54,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:54,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:54,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:54,595 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:54,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,608 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:54,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,608 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,608 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:54,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,609 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:54,609 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:54,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,709 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:54,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:54,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321474718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321474719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321474719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321474719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321474720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:54,734 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:54,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:54,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:54,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:54,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:54,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:54,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:54,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:54,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:54,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:54,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:54,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:54,758 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:54,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:54,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:54,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:54,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:54,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4402df7f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:54,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:54,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:54,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:54,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:54,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:54,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:54,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:54,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:54,769 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:54,769 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:54,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,772 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,776 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,776 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:54,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:54,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:54,776 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,777 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:54,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,777 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,855 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015f03172b850c41ed9218facf2207fd64 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015f03172b850c41ed9218facf2207fd64 2023-04-01 03:56:54,871 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/3f726dcedf064fa9a35faa7e30c0eabd 2023-04-01 03:56:54,877 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:54,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:54,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/a41455a4cedd407890a66553b6e40620 2023-04-01 03:56:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:54,898 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:54,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:54,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:54,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:54,902 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:54,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:54,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:54,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:54,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:54,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:54,918 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:54,918 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:54,918 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:54,918 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:54,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,920 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:54,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6378c4bd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:54,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:54,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,925 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:54,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:54,929 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/192a34e0d24c46558633b79df86b2dd0 2023-04-01 03:56:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:54,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:54,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:54,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:54,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:54,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:54,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:54,935 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,935 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,935 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/3f726dcedf064fa9a35faa7e30c0eabd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd 2023-04-01 03:56:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:54,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:54,937 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:54,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd, entries=150, sequenceid=16, filesize=25.1 K 2023-04-01 03:56:54,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:54,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:54,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:54,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:54,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:54,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:54,941 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:54,941 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,941 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:54,941 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:54,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:54,941 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:54,941 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:54,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:54,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/a41455a4cedd407890a66553b6e40620 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620 2023-04-01 03:56:54,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620, entries=150, sequenceid=16, filesize=11.6 K 2023-04-01 03:56:54,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/192a34e0d24c46558633b79df86b2dd0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0 2023-04-01 03:56:54,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0, entries=150, sequenceid=16, filesize=11.6 K 2023-04-01 03:56:54,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 57b058ac96577c51eb729e1451adecd4 in 572ms, sequenceid=16, compaction requested=false 2023-04-01 03:56:54,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:55,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321414400, max timestamp=1680321415021Num uniques 859; 2023-04-01 03:56:55,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:56:55,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321414400, max timestamp=1680321415021Num uniques 859; 2023-04-01 03:56:55,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:56:55,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:55,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321414400, max timestamp=1680321415021Num uniques 859; 2023-04-01 03:56:55,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:56:55,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=64.85 KB heapSize=184.09 KB 2023-04-01 03:56:55,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:56:55,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:56:55,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:56:55,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,041 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:55,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:55,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:55,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415041, max timestamp=1680321415047Num uniques 859; 2023-04-01 03:56:55,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680321415041, max timestamp=1680321415047Num uniques 859; 2023-04-01 03:56:55,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=7872, suffix heap size=117064,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=172 2023-04-01 03:56:55,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415041, max timestamp=1680321415047Num uniques 859; 2023-04-01 03:56:55,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:56:55,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:56:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:55,062 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:55,062 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:55,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:55,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,064 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:55,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:55,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:55,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:55,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:55,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:55,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:55,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:55,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321415069, max timestamp=1680321415076Num uniques 1031; 2023-04-01 03:56:55,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47226, new segment data size=7688, suffix heap size=135192,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=168 2023-04-01 03:56:55,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415069, max timestamp=1680321415079Num uniques 1009; 2023-04-01 03:56:55,088 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040124db7abe002e4173908230027fc287bb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040124db7abe002e4173908230027fc287bb 2023-04-01 03:56:55,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415069, max timestamp=1680321415079Num uniques 1009; 2023-04-01 03:56:55,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 03:56:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,086 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 03:56:55,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d031367[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:55,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:55,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:55,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:55,095 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:55,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:55,098 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:55,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:55,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:55,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:55,099 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:55,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,106 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,107 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:55,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,107 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:55,108 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:55,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1680321415096, max timestamp=1680321415123Num uniques 1027; 2023-04-01 03:56:55,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-01 03:56:55,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1680321415096, max timestamp=1680321415123Num uniques 1009; 2023-04-01 03:56:55,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=10382, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-04-01 03:56:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415096, max timestamp=1680321415123Num uniques 1009; 2023-04-01 03:56:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 03:56:55,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321475179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680321415160, max timestamp=1680321415179Num uniques 1086; 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680321415160, max timestamp=1680321415179Num uniques 1086; 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680321415160, max timestamp=1680321415179Num uniques 1009; 2023-04-01 03:56:55,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=17676, suffix heap size=132904,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=386 2023-04-01 03:56:55,208 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:55,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:55,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321475211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8929, totalHeapSize=20520, min timestamp=1680321415203, max timestamp=1680321415211Num uniques 1245; 2023-04-01 03:56:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=57019, new segment data size=8929, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-01 03:56:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680321415203, max timestamp=1680321415211Num uniques 1245; 2023-04-01 03:56:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=8930, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-01 03:56:55,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680321415203, max timestamp=1680321415211Num uniques 1245; 2023-04-01 03:56:55,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=57010, new segment data size=8920, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-01 03:56:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:55,227 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:55,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:55,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:55,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,231 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,231 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:55,232 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:55,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:55,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:55,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:55,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:55,238 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,238 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:55,238 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:55,238 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:55,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680321415241, max timestamp=1680321415247Num uniques 1054; 2023-04-01 03:56:55,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-01 03:56:55,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680321415241, max timestamp=1680321415247Num uniques 1054; 2023-04-01 03:56:55,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-01 03:56:55,252 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,253 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680321415241, max timestamp=1680321415247Num uniques 1054; 2023-04-01 03:56:55,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-01 03:56:55,253 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,253 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:55,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@776b5600[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:55,256 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:55,256 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:55,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:55,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:55,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:55,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:55,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:55,288 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321475287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321415267, max timestamp=1680321415287Num uniques 1063; 2023-04-01 03:56:55,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321475288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 03:56:55,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321415267, max timestamp=1680321415287Num uniques 1063; 2023-04-01 03:56:55,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 03:56:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321415267, max timestamp=1680321415287Num uniques 1063; 2023-04-01 03:56:55,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 03:56:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:55,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:55,293 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:55,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:55,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:55,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:55,293 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:55,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,296 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,297 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:55,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,324 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:55,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:55,323 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,324 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10152, totalHeapSize=23328, min timestamp=1680321415348, max timestamp=1680321415355Num uniques 1022; 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46792, new segment data size=10152, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10161, totalHeapSize=23328, min timestamp=1680321415348, max timestamp=1680321415355Num uniques 1022; 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46801, new segment data size=10161, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10155, totalHeapSize=23328, min timestamp=1680321415348, max timestamp=1680321415355Num uniques 1022; 2023-04-01 03:56:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46795, new segment data size=10155, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 03:56:55,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680321415369, max timestamp=1680321415388Num uniques 1081; 2023-04-01 03:56:55,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-01 03:56:55,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680321415369, max timestamp=1680321415388Num uniques 1081; 2023-04-01 03:56:55,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-01 03:56:55,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321415376, max timestamp=1680321415388Num uniques 1081; 2023-04-01 03:56:55,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=6870, suffix heap size=140392,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=150 2023-04-01 03:56:55,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321475409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321475411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680321415403, max timestamp=1680321415416Num uniques 1190; 2023-04-01 03:56:55,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=17860, suffix heap size=151728,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=390 2023-04-01 03:56:55,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680321415403, max timestamp=1680321415416Num uniques 1009; 2023-04-01 03:56:55,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10990, suffix heap size=132904,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=240 2023-04-01 03:56:55,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321475412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680321415403, max timestamp=1680321415416Num uniques 1190; 2023-04-01 03:56:55,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=17860, suffix heap size=151728,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=390 2023-04-01 03:56:55,424 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:55,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:55,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:55,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:55,428 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:55,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:55,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:55,429 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,429 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,430 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:55,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:55,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:55,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:55,434 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,434 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:55,434 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:55,434 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,450 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254086b5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:55,452 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:55,452 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,457 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:55,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:55,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321475459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:55,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:55,465 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:55,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:55,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:55,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:55,465 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:55,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321475464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,473 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:55,473 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,473 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:55,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:55,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:55,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:55,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:55,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:55,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,484 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:55,484 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:55,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:55,484 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,510 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/1f1d985e8a8c4b878b80124e67be35b6 2023-04-01 03:56:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321475522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321475522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321475525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/ad3e8fb4145d44b680558be3c5492064 2023-04-01 03:56:55,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/aeb275038d9b4a36b6b854772e5ebdd1 2023-04-01 03:56:55,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/1f1d985e8a8c4b878b80124e67be35b6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6 2023-04-01 03:56:55,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6, entries=150, sequenceid=38, filesize=25.1 K 2023-04-01 03:56:55,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/ad3e8fb4145d44b680558be3c5492064 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064 2023-04-01 03:56:55,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321475565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064, entries=150, sequenceid=38, filesize=11.6 K 2023-04-01 03:56:55,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/aeb275038d9b4a36b6b854772e5ebdd1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1 2023-04-01 03:56:55,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321475570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1, entries=150, sequenceid=38, filesize=11.6 K 2023-04-01 03:56:55,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 57b058ac96577c51eb729e1451adecd4 in 549ms, sequenceid=38, compaction requested=false 2023-04-01 03:56:55,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:55,584 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:55,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:55,587 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:55,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:55,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:55,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,589 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:55,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:55,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:55,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:55,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:55,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:55,591 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=154.31 KB heapSize=427.78 KB 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:55,605 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040120ed1064686e4954a250eca5d6582fe8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120ed1064686e4954a250eca5d6582fe8 2023-04-01 03:56:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:55,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:55,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321475747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321475747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321475750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321475767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321475775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,778 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:56:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321475851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321475851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:55,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321475853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,013 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/2946ddca31da4c37810697438d4130e3 2023-04-01 03:56:56,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321476055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321476056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321476056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321476069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321476076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321476357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321476359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321476361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,426 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/c83d9df9f6d64c79b3b2e69519c04e47 2023-04-01 03:56:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321476572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321476582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,837 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/1c3ec9037c1b444cb31e9c98dcbeabb4 2023-04-01 03:56:56,842 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/2946ddca31da4c37810697438d4130e3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3 2023-04-01 03:56:56,846 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3, entries=150, sequenceid=229, filesize=25.3 K 2023-04-01 03:56:56,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/c83d9df9f6d64c79b3b2e69519c04e47 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47 2023-04-01 03:56:56,852 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47, entries=150, sequenceid=229, filesize=11.8 K 2023-04-01 03:56:56,853 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/1c3ec9037c1b444cb31e9c98dcbeabb4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4 2023-04-01 03:56:56,862 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4, entries=150, sequenceid=229, filesize=11.8 K 2023-04-01 03:56:56,863 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~427.73 KB/438000, currentSize=33.54 KB/34350 for 57b058ac96577c51eb729e1451adecd4 in 1272ms, sequenceid=229, compaction requested=true 2023-04-01 03:56:56,863 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:56,863 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:56,863 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:56,864 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:56,864 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,866 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:56,866 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:56,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:56,868 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,868 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:56,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:56:56,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:56,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b8e38c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:56,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:56,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:56:56,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:56,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:56:56,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:56,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:56:56,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:56:56,871 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:56,872 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,872 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:56,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:56,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:56,877 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:56,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:56,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:56,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:56,877 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:56,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:56,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:56,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:56,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,893 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:56,893 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:56,893 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:56,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:56,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:56,893 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:56,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:56,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:56,894 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:56,893 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:56,894 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:56,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:56,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321476935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321416931, max timestamp=1680321416936Num uniques 859; 2023-04-01 03:56:56,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:56:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321416931, max timestamp=1680321416936Num uniques 859; 2023-04-01 03:56:56,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:56:56,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321416931, max timestamp=1680321416936Num uniques 859; 2023-04-01 03:56:56,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:56:56,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321476977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321476978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:56,994 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:56,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:56,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:56,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:56,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:56,998 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:56,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:56,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:56,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:56,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:56,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,001 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,001 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,001 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,001 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755550d7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,004 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,004 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,005 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,005 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,005 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,005 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,007 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,009 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,009 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,010 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,014 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,014 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321477041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321477080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321477081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,114 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,118 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,120 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,121 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,123 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,124 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,124 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,124 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58a3d32b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,124 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,126 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,128 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,130 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,130 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,131 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,131 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,131 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,132 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,133 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,136 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,137 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,137 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,237 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,239 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,242 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,242 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,242 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,242 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,242 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,244 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,244 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,244 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797783a4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,245 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,246 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321477247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,248 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,248 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,248 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,248 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,248 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,248 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,250 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,250 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,252 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,252 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,252 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,252 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,252 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,253 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,253 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,253 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,255 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,257 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,259 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,259 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321477282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321477282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,305 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401c06565420bfd4c198f033fad664c6810 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c06565420bfd4c198f033fad664c6810 2023-04-01 03:56:57,359 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,362 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,365 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,365 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,365 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,365 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,366 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,367 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,367 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,367 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,368 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,368 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,368 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,369 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e14bb85[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,372 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,388 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,388 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,388 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,388 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,388 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,390 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,390 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,390 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,393 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,393 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,394 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,394 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,394 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,494 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,497 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,499 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,501 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,501 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,501 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,501 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,503 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,504 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,504 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61188778[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,507 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,507 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,507 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,508 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,509 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,510 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,510 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,512 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,512 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,512 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,513 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,516 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,516 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321477549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321477579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321477585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321477585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:57,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321477587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,616 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,620 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,623 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,623 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,623 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,623 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,623 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,624 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,626 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,627 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,627 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,628 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27642afb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,628 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,629 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,636 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,636 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,638 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,638 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,639 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,646 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,646 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,711 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/058c74d4c0b940469c20672e62035e37 2023-04-01 03:56:57,746 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,749 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,752 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,752 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,752 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,752 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,754 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,755 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624bf710[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,755 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,757 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,757 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,757 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,759 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,761 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,761 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:57,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,767 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,767 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,867 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,870 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,871 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,871 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,872 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:57,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:57,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7918b04f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:57,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:57,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:57,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:57,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:57,883 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:57,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:57,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:57,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:57,884 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:57,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,887 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,888 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:57,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:57,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,889 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:57,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:57,889 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:57,989 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:57,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:57,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:57,993 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:57,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:57,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:57,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:57,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:57,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:57,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:57,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:57,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:57,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:57,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:57,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:57,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:57,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,998 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:57,999 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:57,999 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:58,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c1edb53[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:58,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:58,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:58,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:58,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:58,002 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,002 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:58,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:58,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:58,006 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:58,006 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:58,006 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:58,006 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:58,006 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:58,007 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,007 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:58,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:58,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:58,013 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:58,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,013 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321478056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321478086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:58,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321478089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,113 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:58,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:58,116 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:58,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:58,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:58,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/80c79ab8438d406289ff7747029ca4ae 2023-04-01 03:56:58,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,127 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,127 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,127 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:58,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:58,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:58,128 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,128 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:58,129 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:58,129 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:58,141 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:58,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:58,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aab11d9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:58,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,156 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:58,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:58,156 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:58,157 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:58,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:58,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:58,160 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:58,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:58,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:58,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:58,160 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:58,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:58,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,164 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,164 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:58,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,265 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:58,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:58,268 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:58,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:58,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:58,269 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,269 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,269 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,269 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:58,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:58,271 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,271 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:58,271 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:58,271 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,272 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:58,272 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:58,273 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e58eeb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:58,274 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:58,274 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,277 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,277 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:58,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:58,277 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:58,277 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:58,278 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:58,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:58,280 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:58,280 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:58,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:58,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:58,281 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:58,282 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,282 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,282 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:58,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,286 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:58,286 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:58,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,387 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:58,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:58,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:58,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:58,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:58,390 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:58,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:58,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:58,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,390 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,392 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:58,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:58,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:58,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:58,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:58,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:58,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:58,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:58,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:58,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:58,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:58,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:58,397 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:58,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc3d090[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:58,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:58,399 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:58,401 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,401 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:58,402 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:58,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:58,405 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:58,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:58,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:58,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:58,405 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:58,406 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,406 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,406 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,410 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:58,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:58,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,411 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:58,411 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:58,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:58,511 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:58,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:58,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:58,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:58,513 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:58,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:58,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:58,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:58,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:58,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:56:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:56:58,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:56:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:56:58,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ad4cf8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:56:58,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:56:58,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:56:58,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:56:58,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:56:58,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:56:58,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:56:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:56:58,523 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:56:58,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:56:58,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:56:58,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:56:58,523 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:56:58,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:56:58,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:56:58,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:56:58,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:56:58,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:56:58,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:56:58,533 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:56:58,533 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:56:58,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:56:58,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/e54aa5e4573d4c9487b762a0507677d6 2023-04-01 03:56:58,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/058c74d4c0b940469c20672e62035e37 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37 2023-04-01 03:56:58,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37, entries=300, sequenceid=240, filesize=45.8 K 2023-04-01 03:56:58,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/80c79ab8438d406289ff7747029ca4ae as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae 2023-04-01 03:56:58,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae, entries=150, sequenceid=240, filesize=11.8 K 2023-04-01 03:56:58,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/e54aa5e4573d4c9487b762a0507677d6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6 2023-04-01 03:56:58,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6, entries=150, sequenceid=240, filesize=11.8 K 2023-04-01 03:56:58,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 57b058ac96577c51eb729e1451adecd4 in 1685ms, sequenceid=240, compaction requested=true 2023-04-01 03:56:58,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:58,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:58,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:58,555 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:58,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:58,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:58,556 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124258 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:58,557 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:56:58,557 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47844 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:58,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:56:58,557 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=121.3 K 2023-04-01 03:56:58,557 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,557 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=46.7 K 2023-04-01 03:56:58,557 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 3f726dcedf064fa9a35faa7e30c0eabd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321414335 2023-04-01 03:56:58,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a41455a4cedd407890a66553b6e40620, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321414335 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1f1d985e8a8c4b878b80124e67be35b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680321414405 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ad3e8fb4145d44b680558be3c5492064, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680321414405 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2946ddca31da4c37810697438d4130e3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680321415442 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c83d9df9f6d64c79b3b2e69519c04e47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680321415442 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 058c74d4c0b940469c20672e62035e37, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:56:58,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 80c79ab8438d406289ff7747029ca4ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:56:58,576 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#130 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:58,597 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#129 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:58,617 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/01da5847b07746afa5743a7864d79834 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/01da5847b07746afa5743a7864d79834 2023-04-01 03:56:58,628 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into 01da5847b07746afa5743a7864d79834(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:58,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:58,628 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=12, startTime=1680321418554; duration=0sec 2023-04-01 03:56:58,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:56:58,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:56:58,630 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47844 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:56:58,630 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:56:58,630 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,630 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=46.7 K 2023-04-01 03:56:58,630 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 192a34e0d24c46558633b79df86b2dd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321414335 2023-04-01 03:56:58,631 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting aeb275038d9b4a36b6b854772e5ebdd1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680321414405 2023-04-01 03:56:58,631 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1c3ec9037c1b444cb31e9c98dcbeabb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680321415442 2023-04-01 03:56:58,632 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e54aa5e4573d4c9487b762a0507677d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:56:58,633 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:56:58,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:56:58,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:56:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:56:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:56:58,642 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:56:58,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:56:58,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:56:58,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,653 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#131 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:56:58,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:56:58,657 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:56:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:56:58,657 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:58,657 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:56:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:56:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:56:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:56:58,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:56:58,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:56:58,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:56:58,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:56:58,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:56:58,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:56:58,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:56:58,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:56:58,679 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:56:58,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:56:58,695 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/4129e79d25124a2aafc81220393d6628 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4129e79d25124a2aafc81220393d6628 2023-04-01 03:56:58,705 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into 4129e79d25124a2aafc81220393d6628(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:58,705 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:58,705 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=12, startTime=1680321418555; duration=0sec 2023-04-01 03:56:58,705 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:58,711 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/988ff6fd165941588527f1ebc57a5ed0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/988ff6fd165941588527f1ebc57a5ed0 2023-04-01 03:56:58,715 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into 988ff6fd165941588527f1ebc57a5ed0(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:56:58,715 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:56:58,716 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=12, startTime=1680321418554; duration=0sec 2023-04-01 03:56:58,716 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:56:58,739 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015ce93681aedf41d4ae4f2ec4fe84c458 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015ce93681aedf41d4ae4f2ec4fe84c458 2023-04-01 03:56:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:56:59,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:56:59,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321479094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321479096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321479103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,153 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/781dec769d044c3483aea749551c6bc4 2023-04-01 03:56:59,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321479197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321479400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,564 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/bcc66a57ba674a29bbc44222030269a7 2023-04-01 03:56:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321479598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,615 DEBUG [Thread-899] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:59,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321479604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:56:59,631 DEBUG [Thread-897] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:56:59,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:56:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321479705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,034 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/921879f98c1c404ca4a7016382fa9a83 2023-04-01 03:57:00,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/781dec769d044c3483aea749551c6bc4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4 2023-04-01 03:57:00,049 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4, entries=150, sequenceid=276, filesize=25.4 K 2023-04-01 03:57:00,050 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/bcc66a57ba674a29bbc44222030269a7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7 2023-04-01 03:57:00,056 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7, entries=150, sequenceid=276, filesize=11.9 K 2023-04-01 03:57:00,057 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/921879f98c1c404ca4a7016382fa9a83 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83 2023-04-01 03:57:00,077 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83, entries=150, sequenceid=276, filesize=11.9 K 2023-04-01 03:57:00,078 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 57b058ac96577c51eb729e1451adecd4 in 1399ms, sequenceid=276, compaction requested=false 2023-04-01 03:57:00,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:00,078 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:00,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:00,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730abecd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,084 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,084 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:00,084 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:00,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:00,084 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:00,086 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:00,092 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:00,092 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:00,092 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:00,092 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:00,092 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,101 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:00,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,101 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,201 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:00,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:00,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:00,209 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:00,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:00,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:00,211 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,211 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,212 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,212 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,212 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,212 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:00,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:00,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:00,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:00,222 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:00,222 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,222 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:00,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:00,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:00,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:00,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:00,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:00,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:00,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:00,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:00,228 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38d3e491[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:00,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:00,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:00,236 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:00,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:00,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:00,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:00,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:00,239 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:00,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:00,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:00,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:00,240 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:00,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,244 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,246 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:00,247 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,246 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,247 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,247 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:00,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:00,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,247 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:00,247 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:00,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321480271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,347 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:00,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:00,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:00,357 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:00,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:00,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:00,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,360 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:00,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:00,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:00,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:00,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:00,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:00,363 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,363 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:00,363 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:00,363 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:00,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:00,365 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:00,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d51e44b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:00,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:00,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:00,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:00,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:00,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:00,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:00,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:00,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321480375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:00,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:00,379 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:00,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:00,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:00,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:00,379 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:00,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,385 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,386 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:00,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:00,386 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,386 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:00,386 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:00,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,486 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:00,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:00,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:00,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:00,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:00,490 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:00,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:00,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:00,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,506 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:00,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:00,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:00,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:00,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:00,510 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,510 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:00,510 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:00,510 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6680ad95[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:00,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:00,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:00,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:00,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:00,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:00,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:00,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:00,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:00,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:00,539 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:00,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:00,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:00,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:00,539 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:00,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:00,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321480580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:00,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:00,582 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,582 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,582 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:00,582 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,661 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304017f58e14d683f446c9b2123873bac96a1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017f58e14d683f446c9b2123873bac96a1 2023-04-01 03:57:00,669 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/c01687baff4e4ebd83e4f614037fcc3a 2023-04-01 03:57:00,682 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:00,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:00,693 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:00,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:00,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:00,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6 2023-04-01 03:57:00,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,704 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:00,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:00,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:00,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:00,707 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:00,707 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,707 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:00,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:00,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:00,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:00,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:00,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,713 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:00,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40da8b5e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:00,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:00,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,718 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:00,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:00,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:00,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:00,720 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:00,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:00,724 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:00,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:00,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:00,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:00,724 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:00,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:00,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,730 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:00,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/34e488195e3846d1bf513271ba1a289e 2023-04-01 03:57:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:00,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:00,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:00,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,739 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:00,739 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:00,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:00,739 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:00,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:00,739 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:00,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/c01687baff4e4ebd83e4f614037fcc3a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a 2023-04-01 03:57:00,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a, entries=200, sequenceid=290, filesize=32.3 K 2023-04-01 03:57:00,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6 2023-04-01 03:57:00,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6, entries=150, sequenceid=290, filesize=11.9 K 2023-04-01 03:57:00,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/34e488195e3846d1bf513271ba1a289e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e 2023-04-01 03:57:00,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e, entries=150, sequenceid=290, filesize=11.9 K 2023-04-01 03:57:00,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 57b058ac96577c51eb729e1451adecd4 in 553ms, sequenceid=290, compaction requested=true 2023-04-01 03:57:00,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:00,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:00,766 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:00,766 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:00,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:00,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:00,768 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:00,768 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85166 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:00,768 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:57:00,768 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:57:00,768 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,768 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,768 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/01da5847b07746afa5743a7864d79834, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=35.7 K 2023-04-01 03:57:00,768 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/988ff6fd165941588527f1ebc57a5ed0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=83.2 K 2023-04-01 03:57:00,769 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 988ff6fd165941588527f1ebc57a5ed0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:57:00,769 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 01da5847b07746afa5743a7864d79834, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:57:00,769 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 781dec769d044c3483aea749551c6bc4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321416970 2023-04-01 03:57:00,770 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c01687baff4e4ebd83e4f614037fcc3a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:00,770 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting bcc66a57ba674a29bbc44222030269a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321416970 2023-04-01 03:57:00,770 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 31a6cd4c6ff6485ca799d2c9d5e9c1f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:00,800 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#139 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:00,806 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#138 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:00,823 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/b023553c19f54f74925426c9f25587ad as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/b023553c19f54f74925426c9f25587ad 2023-04-01 03:57:00,839 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:00,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:00,841 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into b023553c19f54f74925426c9f25587ad(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:00,841 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:00,841 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=13, startTime=1680321420766; duration=0sec 2023-04-01 03:57:00,841 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:00,841 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:00,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:00,844 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:00,844 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:57:00,844 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,844 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4129e79d25124a2aafc81220393d6628, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=35.7 K 2023-04-01 03:57:00,844 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4129e79d25124a2aafc81220393d6628, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680321415726 2023-04-01 03:57:00,845 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 921879f98c1c404ca4a7016382fa9a83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321416970 2023-04-01 03:57:00,845 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 34e488195e3846d1bf513271ba1a289e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:00,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:00,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:00,860 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:00,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:00,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:00,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,862 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:00,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:00,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:00,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:00,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:00,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:00,866 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#140 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:00,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:00,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:00,869 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:00,869 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:00,869 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:00,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:00,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:00,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:00,874 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:00,874 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:00,875 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:00,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:00,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:00,922 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012be7459f2f384c889c27f5ec4a372db1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012be7459f2f384c889c27f5ec4a372db1 2023-04-01 03:57:00,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:00,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321480924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44178 deadline: 1680321481026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:01,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321481113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,118 DEBUG [Thread-893] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:01,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:01,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321481123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,127 DEBUG [Thread-891] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:01,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321481230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,310 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/a82f18fa97194925b5e5d045f3ddded6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/a82f18fa97194925b5e5d045f3ddded6 2023-04-01 03:57:01,321 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into a82f18fa97194925b5e5d045f3ddded6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:01,321 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:01,321 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=13, startTime=1680321420767; duration=0sec 2023-04-01 03:57:01,321 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:01,336 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/fee8398b21a14befa01b439a4cdcaab3 2023-04-01 03:57:01,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321481531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,747 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/869fcc1b83f844228c56523566417bef 2023-04-01 03:57:01,759 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/829127184ad840bcaf4fd2534bf98750 2023-04-01 03:57:01,764 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/fee8398b21a14befa01b439a4cdcaab3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3 2023-04-01 03:57:01,768 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3, entries=150, sequenceid=311, filesize=25.4 K 2023-04-01 03:57:01,769 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/869fcc1b83f844228c56523566417bef as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef 2023-04-01 03:57:01,773 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef, entries=150, sequenceid=311, filesize=11.9 K 2023-04-01 03:57:01,775 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/829127184ad840bcaf4fd2534bf98750 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750 2023-04-01 03:57:01,779 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750, entries=150, sequenceid=311, filesize=11.9 K 2023-04-01 03:57:01,779 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 57b058ac96577c51eb729e1451adecd4 in 905ms, sequenceid=311, compaction requested=false 2023-04-01 03:57:01,779 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:01,779 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:01,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:01,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:01,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:01,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:01,781 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,781 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:01,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:01,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:01,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:01,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@670a08b8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:01,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:01,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,785 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:01,785 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,785 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:01,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:01,789 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:01,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:01,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:01,791 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:01,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:01,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:01,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:01,791 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:01,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:01,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:01,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:01,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:01,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:01,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:01,795 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,795 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:01,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:01,895 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:01,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:01,900 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:01,902 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:01,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:01,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:01,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:01,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:01,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:01,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:01,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:01,906 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:01,906 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:01,906 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:01,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:01,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:01,914 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:01,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:01,915 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:01,915 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:02,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:02,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:02,055 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/d0a0718efe874635b97510f4204f0101 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d0a0718efe874635b97510f4204f0101 2023-04-01 03:57:02,059 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into d0a0718efe874635b97510f4204f0101(size=25.7 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:57:02,059 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:02,059 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=13, startTime=1680321420766; duration=1sec 2023-04-01 03:57:02,059 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:02,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:02,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321482083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:02,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:02,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321482187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:02,345 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304014e587178c94f4f6d8e950c896f8edf3e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014e587178c94f4f6d8e950c896f8edf3e 2023-04-01 03:57:02,350 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/22facf1421b845ae980a51b62caf91cd 2023-04-01 03:57:02,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321482388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:02,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:02,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321482690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:02,761 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/eb0b422ba22e4d248f141b8eaf6be80b 2023-04-01 03:57:03,171 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/50cc650fffa54be3818d9592a63383a4 2023-04-01 03:57:03,176 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/22facf1421b845ae980a51b62caf91cd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd 2023-04-01 03:57:03,181 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd, entries=150, sequenceid=324, filesize=25.4 K 2023-04-01 03:57:03,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/eb0b422ba22e4d248f141b8eaf6be80b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b 2023-04-01 03:57:03,187 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b, entries=150, sequenceid=324, filesize=11.9 K 2023-04-01 03:57:03,188 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/50cc650fffa54be3818d9592a63383a4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4 2023-04-01 03:57:03,192 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4, entries=150, sequenceid=324, filesize=11.9 K 2023-04-01 03:57:03,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321483192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,193 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 57b058ac96577c51eb729e1451adecd4 in 1278ms, sequenceid=324, compaction requested=true 2023-04-01 03:57:03,193 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:03,193 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:03,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:03,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:03,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:03,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:03,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:03,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:03,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@158bdbae[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:03,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:03,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:03,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:03,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,199 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:03,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:03,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:03,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:03,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:03,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:03,202 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:03,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:03,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:03,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:03,202 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:03,203 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,203 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:03,203 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,203 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,205 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:03,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:03,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:03,207 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:03,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:03,207 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:03,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:03,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:03,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:03,307 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:03,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:03,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:03,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:03,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:03,310 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:03,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:03,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:03,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:03,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:03,314 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:03,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:03,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:03,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:03,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:03,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:03,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:03,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:03,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:03,320 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:03,320 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:03,320 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:57:03,320 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:03,320 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,321 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:03,321 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,321 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:03,321 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,348 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304019684dfa5fa4149fa8bcf2c26bf35a9fc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019684dfa5fa4149fa8bcf2c26bf35a9fc 2023-04-01 03:57:03,353 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5074e1ccd8b844aea6adb88b896ed6b7 2023-04-01 03:57:03,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:03,642 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:03,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:03,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321483658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:03,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321483661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,762 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/1f51172194d445adaf6784ec6080d81e 2023-04-01 03:57:03,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:03,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321483762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:03,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321483765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,800 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f7193abaa13d4f969cae439406daecc5 2023-04-01 03:57:03,804 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5074e1ccd8b844aea6adb88b896ed6b7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7 2023-04-01 03:57:03,808 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7, entries=150, sequenceid=345, filesize=25.4 K 2023-04-01 03:57:03,808 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/1f51172194d445adaf6784ec6080d81e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e 2023-04-01 03:57:03,814 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e, entries=150, sequenceid=345, filesize=11.9 K 2023-04-01 03:57:03,815 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f7193abaa13d4f969cae439406daecc5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5 2023-04-01 03:57:03,820 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5, entries=150, sequenceid=345, filesize=11.9 K 2023-04-01 03:57:03,821 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 57b058ac96577c51eb729e1451adecd4 in 501ms, sequenceid=345, compaction requested=true 2023-04-01 03:57:03,821 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:03,821 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:03,821 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:03,821 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:03,821 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,822 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:03,822 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:03,824 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7604308f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:03,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:03,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:03,829 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:03,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:03,832 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:03,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:03,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:03,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:03,833 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:03,833 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,834 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:03,834 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:03,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:03,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,835 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,837 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:03,837 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:03,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:03,837 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:03,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:03,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:03,837 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:03,838 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:03,837 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:03,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:03,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:03,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:03,938 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:03,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:03,941 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:03,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:03,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:03,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:03,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:03,942 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:03,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:03,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:03,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:03,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:03,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:03,944 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:03,944 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:03,957 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401cccf0de61f9f41f3ad5626f7989e16b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cccf0de61f9f41f3ad5626f7989e16b0 2023-04-01 03:57:03,962 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/8af57e902c6c4b8caf1fec518a37f211 2023-04-01 03:57:03,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:03,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:03,976 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/8a5296d27dfe48c8abcc79c63932f1c0 2023-04-01 03:57:04,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321483999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321484102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321484198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321484306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,388 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/6f2f6c7d439e473a8290d60d8e889793 2023-04-01 03:57:04,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/8af57e902c6c4b8caf1fec518a37f211 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211 2023-04-01 03:57:04,397 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211, entries=150, sequenceid=357, filesize=25.4 K 2023-04-01 03:57:04,398 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/8a5296d27dfe48c8abcc79c63932f1c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0 2023-04-01 03:57:04,401 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0, entries=150, sequenceid=357, filesize=11.9 K 2023-04-01 03:57:04,402 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/6f2f6c7d439e473a8290d60d8e889793 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793 2023-04-01 03:57:04,406 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793, entries=150, sequenceid=357, filesize=11.9 K 2023-04-01 03:57:04,407 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 57b058ac96577c51eb729e1451adecd4 in 463ms, sequenceid=357, compaction requested=true 2023-04-01 03:57:04,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:04,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,410 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:04,410 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:04,411 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,411 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:04,411 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,412 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6290c56a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:04,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:04,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:04,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:04,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:04,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:04,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:04,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:04,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:04,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:04,419 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:04,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:04,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:04,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:04,419 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:04,421 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,421 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:04,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,422 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:04,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:04,422 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,423 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:04,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:04,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:04,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:04,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:04,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:04,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:04,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:04,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:04,428 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:04,428 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:04,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:04,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:04,528 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:04,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:04,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:04,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:04,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:04,531 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:04,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:04,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:04,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:04,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:04,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:04,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:04,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:04,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:04,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:04,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:04,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:04,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:04,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:04,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:04,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:04,534 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:04,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:04,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:04,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:04,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321484620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321484723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321484925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:04,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321484925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:04,947 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304019e4b262c59234462b263167b5008d621 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e4b262c59234462b263167b5008d621 2023-04-01 03:57:05,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44208 deadline: 1680321485126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,129 DEBUG [Thread-893] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:05,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44228 deadline: 1680321485149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,152 DEBUG [Thread-891] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:05,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321485228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321485230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,352 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/29d2a36a68fa4677a08287ecb8cc07b9 2023-04-01 03:57:05,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321485733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:05,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321485735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:05,764 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/68400618ab74462a8e815a06579c8a38 2023-04-01 03:57:06,173 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d1b3d01c037349dbb76ec241f8bef394 2023-04-01 03:57:06,177 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/29d2a36a68fa4677a08287ecb8cc07b9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9 2023-04-01 03:57:06,180 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9, entries=150, sequenceid=377, filesize=25.4 K 2023-04-01 03:57:06,181 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/68400618ab74462a8e815a06579c8a38 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38 2023-04-01 03:57:06,184 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38, entries=150, sequenceid=377, filesize=11.9 K 2023-04-01 03:57:06,185 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d1b3d01c037349dbb76ec241f8bef394 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394 2023-04-01 03:57:06,189 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394, entries=150, sequenceid=377, filesize=11.9 K 2023-04-01 03:57:06,189 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 57b058ac96577c51eb729e1451adecd4 in 1655ms, sequenceid=377, compaction requested=true 2023-04-01 03:57:06,189 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:06,189 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55684e48[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,194 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,201 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,201 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,201 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,201 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,201 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:06,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:06,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:06,219 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,219 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,219 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,220 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,221 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,226 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401c4f6ef50e21b4235832c36d274013ccf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c4f6ef50e21b4235832c36d274013ccf 2023-04-01 03:57:06,237 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/f5134dacdefe48069022612508d4608e 2023-04-01 03:57:06,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,242 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,242 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,242 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:06,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321486280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,342 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:06,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:06,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:06,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:06,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:06,345 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:06,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:06,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:06,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,349 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:06,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:06,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:06,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:06,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:06,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:06,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:06,350 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,350 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:06,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:06,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dfd4522[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,373 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,373 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,373 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321486385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,397 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,397 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,397 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,397 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,397 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,398 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,398 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,398 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,399 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,401 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,425 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,425 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,425 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,526 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:06,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:06,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:06,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:06,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:06,528 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:06,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:06,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:06,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:06,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:06,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:06,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:06,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:06,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:06,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:06,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:06,532 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,532 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:06,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:06,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7054db[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,541 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,541 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,549 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,549 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:06,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321486590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/9be14d2065684a41b01f24a6bc97d3b2 2023-04-01 03:57:06,650 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:06,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:06,652 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:06,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:06,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:06,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,662 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:06,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:06,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:06,663 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,663 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:06,663 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:06,663 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,679 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@979d0dc[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,687 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,688 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,688 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,688 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,690 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,690 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,691 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,691 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,693 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,694 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:06,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:06,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321486738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321486746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,794 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:06,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:06,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:06,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:06,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:06,797 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:06,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:06,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:06,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:06,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:06,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:06,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:06,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:06,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:06,801 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,801 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:06,801 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:06,801 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,803 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5396ce20[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,816 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,816 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:06,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:06,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321486894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,916 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:06,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:06,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:06,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:06,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:06,919 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:06,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:06,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:06,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,920 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:06,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:06,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:06,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:06,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:06,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:06,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:06,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14854525[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:06,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:06,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:06,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:06,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:06,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:06,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:06,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:06,928 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:06,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:06,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:06,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:06,928 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:06,929 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,929 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:06,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,929 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:06,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:06,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:06,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:06,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:06,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:06,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:06,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:06,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:06,933 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:06,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:06,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:06,933 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:07,033 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:07,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:07,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:07,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:07,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:07,036 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:07,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:07,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:07,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:07,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,041 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:07,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:07,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:07,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:07,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:07,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:07,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:07,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:07,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:07,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:07,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:07,042 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,042 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:07,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:07,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:07,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:07,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:07,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:07,044 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:07,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:07,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:07,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6b0de2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:07,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:07,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,046 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:07,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:07,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:07,046 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,046 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:07,047 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:07,047 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:07,047 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,047 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:07,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:07,049 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:07,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:07,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:07,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:07,049 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:07,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:07,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:07,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:07,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:07,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:07,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:07,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:07,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:07,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:07,053 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:07,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,053 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:07,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:07,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:07,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f4bdd149309a47918985c09007d4fe1e 2023-04-01 03:57:07,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/f5134dacdefe48069022612508d4608e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e 2023-04-01 03:57:07,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e, entries=200, sequenceid=390, filesize=32.3 K 2023-04-01 03:57:07,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/9be14d2065684a41b01f24a6bc97d3b2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2 2023-04-01 03:57:07,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2, entries=150, sequenceid=390, filesize=11.9 K 2023-04-01 03:57:07,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f4bdd149309a47918985c09007d4fe1e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e 2023-04-01 03:57:07,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e, entries=150, sequenceid=390, filesize=11.9 K 2023-04-01 03:57:07,077 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 57b058ac96577c51eb729e1451adecd4 in 869ms, sequenceid=390, compaction requested=true 2023-04-01 03:57:07,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:07,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:07,077 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 03:57:07,077 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 03:57:07,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:07,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:07,079 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 189602 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 03:57:07,079 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85542 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 03:57:07,079 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:57:07,079 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:57:07,079 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,079 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,079 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/b023553c19f54f74925426c9f25587ad, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=83.5 K 2023-04-01 03:57:07,079 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d0a0718efe874635b97510f4204f0101, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=185.2 K 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b023553c19f54f74925426c9f25587ad, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d0a0718efe874635b97510f4204f0101, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 869fcc1b83f844228c56523566417bef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680321420257 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting fee8398b21a14befa01b439a4cdcaab3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680321420257 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting eb0b422ba22e4d248f141b8eaf6be80b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680321420903 2023-04-01 03:57:07,080 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 22facf1421b845ae980a51b62caf91cd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680321420903 2023-04-01 03:57:07,081 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1f51172194d445adaf6784ec6080d81e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680321422075 2023-04-01 03:57:07,081 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5074e1ccd8b844aea6adb88b896ed6b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680321422075 2023-04-01 03:57:07,081 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8a5296d27dfe48c8abcc79c63932f1c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680321423648 2023-04-01 03:57:07,081 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 8af57e902c6c4b8caf1fec518a37f211, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680321423648 2023-04-01 03:57:07,081 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 68400618ab74462a8e815a06579c8a38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680321423990 2023-04-01 03:57:07,082 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 29d2a36a68fa4677a08287ecb8cc07b9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680321423990 2023-04-01 03:57:07,082 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9be14d2065684a41b01f24a6bc97d3b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:07,082 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f5134dacdefe48069022612508d4608e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:07,093 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#159 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:07,102 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#160 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:07,112 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/c1e6cd87a9b64fd6ab1b274ea8c28b03 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c1e6cd87a9b64fd6ab1b274ea8c28b03 2023-04-01 03:57:07,118 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into c1e6cd87a9b64fd6ab1b274ea8c28b03(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:07,118 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:07,118 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=9, startTime=1680321427077; duration=0sec 2023-04-01 03:57:07,118 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:07,118 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 03:57:07,120 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85542 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 03:57:07,120 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:57:07,120 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,120 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/a82f18fa97194925b5e5d045f3ddded6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=83.5 K 2023-04-01 03:57:07,121 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a82f18fa97194925b5e5d045f3ddded6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680321419077 2023-04-01 03:57:07,121 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 829127184ad840bcaf4fd2534bf98750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680321420257 2023-04-01 03:57:07,121 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 50cc650fffa54be3818d9592a63383a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680321420903 2023-04-01 03:57:07,122 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f7193abaa13d4f969cae439406daecc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680321422075 2023-04-01 03:57:07,122 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6f2f6c7d439e473a8290d60d8e889793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680321423648 2023-04-01 03:57:07,122 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d1b3d01c037349dbb76ec241f8bef394, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680321423990 2023-04-01 03:57:07,122 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f4bdd149309a47918985c09007d4fe1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:07,131 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#161 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:07,154 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:07,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:07,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:07,156 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:07,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:07,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:07,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:07,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:07,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:07,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:07,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:07,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:07,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:07,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:07,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:07,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:07,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:07,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:07,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:07,159 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:07,159 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:07,159 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:07,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:07,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:07,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:07,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321487426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:07,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321487532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,543 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/59d51c9034524200ba12d46881071d3c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/59d51c9034524200ba12d46881071d3c 2023-04-01 03:57:07,547 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into 59d51c9034524200ba12d46881071d3c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:07,547 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:07,547 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=9, startTime=1680321427077; duration=0sec 2023-04-01 03:57:07,547 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:07,571 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ccf67591dcf44fe698d6fa4fbaa3329f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ccf67591dcf44fe698d6fa4fbaa3329f 2023-04-01 03:57:07,576 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/4a1d09abef644fa3958520c869aef266 2023-04-01 03:57:07,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:07,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321487735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:07,986 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/7c1586b504b14da59fbf80c4c7f1b7a6 2023-04-01 03:57:08,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321488039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,333 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/a8d6ec8091584273b7a75c3b1740b191 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/a8d6ec8091584273b7a75c3b1740b191 2023-04-01 03:57:08,340 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into a8d6ec8091584273b7a75c3b1740b191(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:57:08,340 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:08,340 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=9, startTime=1680321427077; duration=1sec 2023-04-01 03:57:08,340 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:08,397 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/adc6fb6f1ced410ea22ec0b4d4aea9ea 2023-04-01 03:57:08,402 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/4a1d09abef644fa3958520c869aef266 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266 2023-04-01 03:57:08,406 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266, entries=150, sequenceid=410, filesize=25.4 K 2023-04-01 03:57:08,407 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/7c1586b504b14da59fbf80c4c7f1b7a6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6 2023-04-01 03:57:08,412 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6, entries=150, sequenceid=410, filesize=11.9 K 2023-04-01 03:57:08,414 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/adc6fb6f1ced410ea22ec0b4d4aea9ea as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea 2023-04-01 03:57:08,418 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea, entries=150, sequenceid=410, filesize=11.9 K 2023-04-01 03:57:08,419 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 57b058ac96577c51eb729e1451adecd4 in 1260ms, sequenceid=410, compaction requested=false 2023-04-01 03:57:08,419 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:08,419 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,419 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:08,419 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:08,419 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,421 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:08,421 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:08,423 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:08,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bd3d783[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:08,424 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:08,424 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:08,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:08,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:08,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:08,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:08,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:08,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:08,430 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:08,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:08,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:08,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:08,431 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:08,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:08,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:08,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:08,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,453 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:08,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,453 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:08,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:08,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:08,554 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:08,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:08,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:08,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:08,556 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:08,556 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,556 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:08,556 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:08,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:08,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:08,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:08,559 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,559 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:08,559 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:08,559 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:08,560 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:08,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,561 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,562 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,562 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:08,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:08,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed1e317[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:08,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,565 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:08,565 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:08,565 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:08,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:08,567 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:08,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:08,569 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:08,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:08,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:08,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:08,569 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:08,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,571 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,572 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:08,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:08,574 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,575 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:08,575 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:08,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321488578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,675 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:08,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:08,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:08,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:08,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:08,678 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:08,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:08,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:08,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,680 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:08,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:08,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:08,681 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,681 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:08,681 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:08,681 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:08,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:08,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:08,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:08,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321488681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:08,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:08,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c32385[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:08,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,687 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:08,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:08,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:08,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:08,688 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:08,689 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:08,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:08,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:08,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:08,690 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:08,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,691 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:08,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:08,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,694 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:08,694 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:08,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321488751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,755 DEBUG [Thread-899] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:08,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44254 deadline: 1680321488760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,762 DEBUG [Thread-897] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:08,794 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:08,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:08,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:08,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:08,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:08,797 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:08,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:08,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:08,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:08,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:08,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:08,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:08,800 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,800 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:08,800 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:08,800 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:08,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:08,802 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:08,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44332c62[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:08,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:08,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,806 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:08,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:08,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:08,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:08,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:08,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:08,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:08,809 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:08,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:08,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:08,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:08,810 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:08,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,815 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:08,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,815 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:08,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,815 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:08,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:08,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321488885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,916 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:08,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:08,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:08,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:08,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:08,918 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:08,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:08,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:08,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,921 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:08,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:08,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:08,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:08,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:08,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:08,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:08,923 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:08,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:08,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312984a8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:08,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,925 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:08,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:08,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:08,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,926 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:08,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:08,927 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:08,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:08,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:08,927 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:08,927 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:08,928 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,928 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,928 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:08,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:08,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:08,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:08,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:08,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:08,931 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,931 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:08,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:08,931 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:08,957 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040183a30e2d61c5416ea573bfaecfa627c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040183a30e2d61c5416ea573bfaecfa627c0 2023-04-01 03:57:09,031 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,035 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,035 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,035 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,035 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,035 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,035 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,037 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,037 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,037 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,037 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,038 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,038 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,038 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,038 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,039 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,039 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,039 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306110c1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,040 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,040 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,041 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,041 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,041 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,041 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,042 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,044 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,044 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,045 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,045 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,045 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,079 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,080 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,080 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,180 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,182 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,182 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,182 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,182 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,185 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,187 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,189 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,189 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1917d243[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,189 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,189 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,190 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:09,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321489188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,194 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,194 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,199 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,200 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:09,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,300 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,302 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449abd3f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,309 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,315 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,315 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,316 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,318 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,319 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,319 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,319 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,319 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,362 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/e7701ca422ab4b31a163bc677528d80a 2023-04-01 03:57:09,419 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,421 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,422 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,422 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,422 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,429 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,430 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,430 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,431 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,431 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,431 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,431 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39eb6dc0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,456 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,457 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,458 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,458 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,458 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,461 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,461 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,561 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,564 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,566 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,566 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,566 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,567 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,568 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,568 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,568 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,568 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b48524[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,571 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,571 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,572 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,572 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,572 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,573 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,574 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,575 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,575 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,577 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,581 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,581 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,581 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,681 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,684 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,687 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,688 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,688 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,688 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,688 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,689 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,689 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,689 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,689 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2696ca9b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,694 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,694 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,694 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,694 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321489694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,696 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,698 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,698 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,698 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,698 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,698 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,699 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,699 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,699 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,700 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:09,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,703 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,704 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,704 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,704 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,704 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/7fe2438c9dfd42a1a7eeb5610de023c2 2023-04-01 03:57:09,804 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,806 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,809 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,811 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,811 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,812 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,812 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,813 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f4e833[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,818 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,818 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,818 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,818 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,820 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,820 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,820 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,820 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,820 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,822 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,823 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,828 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,828 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:09,828 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,828 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,828 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,828 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:09,928 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:09,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:09,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:09,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:09,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:09,931 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:09,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:09,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:09,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,940 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:09,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:09,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:09,942 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,942 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:09,942 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:09,942 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,943 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:09,943 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:09,954 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:09,956 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:09,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55931660[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:09,956 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,962 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:09,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:09,963 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,963 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:09,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:09,963 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:09,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:09,966 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:09,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:09,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:09,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:09,966 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:09,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:09,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:09,967 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,968 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:09,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,970 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:09,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:09,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:09,971 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:09,971 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:09,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:09,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:09,971 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:09,971 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:10,072 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:10,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:10,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:10,074 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:10,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:10,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:10,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,076 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,076 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:10,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:10,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:10,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:10,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:10,080 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,080 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:10,080 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:10,080 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:10,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:10,082 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:10,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ab2453[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:10,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:10,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,085 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:10,085 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:10,085 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:10,086 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,086 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:10,087 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,087 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,087 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,087 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:10,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:10,089 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:10,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:10,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:10,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:10,089 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:10,091 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,091 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,091 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,091 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,092 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,096 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:10,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:10,097 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,097 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:10,097 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:10,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/9dfff0049b894c70a1b8d6fc282d421f 2023-04-01 03:57:10,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/e7701ca422ab4b31a163bc677528d80a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a 2023-04-01 03:57:10,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a, entries=200, sequenceid=426, filesize=32.3 K 2023-04-01 03:57:10,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/7fe2438c9dfd42a1a7eeb5610de023c2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2 2023-04-01 03:57:10,197 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:10,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:10,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:10,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2, entries=150, sequenceid=426, filesize=11.9 K 2023-04-01 03:57:10,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/9dfff0049b894c70a1b8d6fc282d421f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f 2023-04-01 03:57:10,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:10,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:10,202 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:10,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:10,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:10,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f, entries=150, sequenceid=426, filesize=11.9 K 2023-04-01 03:57:10,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 57b058ac96577c51eb729e1451adecd4 in 1662ms, sequenceid=426, compaction requested=true 2023-04-01 03:57:10,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:10,206 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:10,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:10,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 03:57:10,207 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85656 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:57:10,208 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,208 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,208 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c1e6cd87a9b64fd6ab1b274ea8c28b03, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=36.2 K 2023-04-01 03:57:10,208 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/a8d6ec8091584273b7a75c3b1740b191, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=83.6 K 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c1e6cd87a9b64fd6ab1b274ea8c28b03, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:10,208 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting a8d6ec8091584273b7a75c3b1740b191, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:10,209 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7c1586b504b14da59fbf80c4c7f1b7a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680321426258 2023-04-01 03:57:10,209 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4a1d09abef644fa3958520c869aef266, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680321426258 2023-04-01 03:57:10,209 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7fe2438c9dfd42a1a7eeb5610de023c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427422 2023-04-01 03:57:10,209 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e7701ca422ab4b31a163bc677528d80a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427417 2023-04-01 03:57:10,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:10,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:10,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:10,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:10,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:10,212 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:10,212 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,219 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:10,229 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#168 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:10,260 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/3175d79c2d6b460d97c463e6cff01638 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/3175d79c2d6b460d97c463e6cff01638 2023-04-01 03:57:10,266 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into 3175d79c2d6b460d97c463e6cff01638(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:10,266 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,266 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=13, startTime=1680321430206; duration=0sec 2023-04-01 03:57:10,266 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040164ee24b309564cfb84f26f4abd64450e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040164ee24b309564cfb84f26f4abd64450e 2023-04-01 03:57:10,266 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:10,266 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:10,268 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:10,268 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:57:10,268 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,268 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/59d51c9034524200ba12d46881071d3c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=36.2 K 2023-04-01 03:57:10,269 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 59d51c9034524200ba12d46881071d3c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680321424612 2023-04-01 03:57:10,269 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting adc6fb6f1ced410ea22ec0b4d4aea9ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680321426258 2023-04-01 03:57:10,269 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9dfff0049b894c70a1b8d6fc282d421f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427422 2023-04-01 03:57:10,284 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:10,286 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=444, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/645d4e6e0e80400fab6991728f4f085e 2023-04-01 03:57:10,306 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/10faae8ca9d144c5b67eb3c1f8d6c8a1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/10faae8ca9d144c5b67eb3c1f8d6c8a1 2023-04-01 03:57:10,307 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/98c3309edc3b473aa99ec69591b3e187 2023-04-01 03:57:10,312 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into 10faae8ca9d144c5b67eb3c1f8d6c8a1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:10,312 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,312 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=13, startTime=1680321430206; duration=0sec 2023-04-01 03:57:10,312 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:10,331 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/4098f61ebff543d9bea1aaf5b16cea15 2023-04-01 03:57:10,336 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/645d4e6e0e80400fab6991728f4f085e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e 2023-04-01 03:57:10,341 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e, entries=150, sequenceid=444, filesize=25.4 K 2023-04-01 03:57:10,342 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/98c3309edc3b473aa99ec69591b3e187 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187 2023-04-01 03:57:10,347 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187, entries=150, sequenceid=444, filesize=11.9 K 2023-04-01 03:57:10,348 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/4098f61ebff543d9bea1aaf5b16cea15 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15 2023-04-01 03:57:10,359 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15, entries=150, sequenceid=444, filesize=11.9 K 2023-04-01 03:57:10,360 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 57b058ac96577c51eb729e1451adecd4 in 148ms, sequenceid=444, compaction requested=false 2023-04-01 03:57:10,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:10,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:10,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:10,360 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:10,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6079eacf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:10,364 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:10,364 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,365 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:10,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:10,367 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:10,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:10,370 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:10,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:10,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:10,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:10,370 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:10,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,372 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,373 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:10,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:10,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,376 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:10,376 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:10,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,476 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:10,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:10,480 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:10,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:10,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:10,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,493 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:10,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:10,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:10,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:10,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:10,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:10,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:10,495 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,495 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:10,495 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,495 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:10,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:10,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:10,496 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:10,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,504 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,505 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,505 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:10,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70dc3383[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:10,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:10,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:10,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:10,522 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:10,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:10,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:10,525 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:10,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:10,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:10,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:10,525 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:10,526 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:10,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:10,532 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:10,532 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:10,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,632 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:10,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:10,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:10,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:10,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:10,636 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:10,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:10,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:10,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,637 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,637 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:10,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:10,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:10,640 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:10,640 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:10,640 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,640 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:10,640 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:10,640 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:10,643 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d311bee[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:10,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:10,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:10,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:10,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:10,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:10,648 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:10,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:10,651 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:10,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:10,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:10,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:10,651 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:10,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:10,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:10,657 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:10,657 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:10,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:10,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,658 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:10,658 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:10,758 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:10,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:10,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:10,761 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:10,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:10,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:10,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:10,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:10,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:10,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:10,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:10,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:10,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:10,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:10,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:10,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:10,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:10,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:10,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:10,766 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:10,766 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:10,766 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-01 03:57:10,766 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:10,766 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,767 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:10,767 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,767 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:10,767 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:10,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:10,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:10,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321430861, max timestamp=1680321430903Num uniques 859; 2023-04-01 03:57:10,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:57:10,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321430861, max timestamp=1680321430903Num uniques 859; 2023-04-01 03:57:10,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:57:10,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321430861, max timestamp=1680321430903Num uniques 859; 2023-04-01 03:57:10,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:57:10,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680321430945, max timestamp=1680321430973Num uniques 1068; 2023-04-01 03:57:10,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680321430945, max timestamp=1680321430973Num uniques 1068; 2023-04-01 03:57:10,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321430948, max timestamp=1680321430973Num uniques 1068; 2023-04-01 03:57:10,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 03:57:10,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 03:57:10,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680321430994, max timestamp=1680321431022Num uniques 1127; 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680321430994, max timestamp=1680321431022Num uniques 1127; 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680321430994, max timestamp=1680321431022Num uniques 1027; 2023-04-01 03:57:11,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-01 03:57:11,090 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/21e068288f884e2f8f1dba1a567cfa9c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/21e068288f884e2f8f1dba1a567cfa9c 2023-04-01 03:57:11,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680321431069, max timestamp=1680321431091Num uniques 1086; 2023-04-01 03:57:11,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-01 03:57:11,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680321431069, max timestamp=1680321431091Num uniques 1086; 2023-04-01 03:57:11,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-01 03:57:11,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680321431069, max timestamp=1680321431091Num uniques 1086; 2023-04-01 03:57:11,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-01 03:57:11,101 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into 21e068288f884e2f8f1dba1a567cfa9c(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:11,101 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:11,101 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=13, startTime=1680321430206; duration=0sec 2023-04-01 03:57:11,102 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:11,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321431137, max timestamp=1680321431154Num uniques 1145; 2023-04-01 03:57:11,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-01 03:57:11,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321431137, max timestamp=1680321431154Num uniques 1145; 2023-04-01 03:57:11,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-01 03:57:11,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321431137, max timestamp=1680321431154Num uniques 1095; 2023-04-01 03:57:11,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-01 03:57:11,212 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040199f25b0e7d2d483594280657e4dae3c9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040199f25b0e7d2d483594280657e4dae3c9 2023-04-01 03:57:11,219 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=453, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/796c947cae1f437990ccb148fc9d43b6 2023-04-01 03:57:11,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680321431194, max timestamp=1680321431221Num uniques 1154; 2023-04-01 03:57:11,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-01 03:57:11,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680321431197, max timestamp=1680321431221Num uniques 1154; 2023-04-01 03:57:11,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-01 03:57:11,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321431197, max timestamp=1680321431221Num uniques 1154; 2023-04-01 03:57:11,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=6870, suffix heap size=147984,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=150 2023-04-01 03:57:11,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680321431253, max timestamp=1680321431274Num uniques 1063; 2023-04-01 03:57:11,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-01 03:57:11,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680321431253, max timestamp=1680321431274Num uniques 1009; 2023-04-01 03:57:11,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-04-01 03:57:11,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680321431253, max timestamp=1680321431274Num uniques 1013; 2023-04-01 03:57:11,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680321431303, max timestamp=1680321431323Num uniques 1072; 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680321431303, max timestamp=1680321431323Num uniques 1072; 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680321431303, max timestamp=1680321431323Num uniques 1072; 2023-04-01 03:57:11,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-01 03:57:11,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321431372, max timestamp=1680321431382Num uniques 1031; 2023-04-01 03:57:11,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:57:11,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321431372, max timestamp=1680321431382Num uniques 1031; 2023-04-01 03:57:11,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:57:11,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321431372, max timestamp=1680321431382Num uniques 1031; 2023-04-01 03:57:11,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:57:11,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680321431410, max timestamp=1680321431430Num uniques 1040; 2023-04-01 03:57:11,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-01 03:57:11,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680321431410, max timestamp=1680321431430Num uniques 1040; 2023-04-01 03:57:11,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-01 03:57:11,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680321431410, max timestamp=1680321431430Num uniques 1040; 2023-04-01 03:57:11,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680321431482, max timestamp=1680321431495Num uniques 1199; 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680321431482, max timestamp=1680321431495Num uniques 1199; 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680321431482, max timestamp=1680321431495Num uniques 1199; 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-01 03:57:11,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-01 03:57:11,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680321431524, max timestamp=1680321431555Num uniques 1058; 2023-04-01 03:57:11,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-01 03:57:11,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680321431524, max timestamp=1680321431555Num uniques 1058; 2023-04-01 03:57:11,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680321431524, max timestamp=1680321431555Num uniques 1058; 2023-04-01 03:57:11,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-01 03:57:11,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-01 03:57:11,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680321431578, max timestamp=1680321431618Num uniques 1067; 2023-04-01 03:57:11,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-01 03:57:11,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680321431578, max timestamp=1680321431618Num uniques 1017; 2023-04-01 03:57:11,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-01 03:57:11,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680321431578, max timestamp=1680321431618Num uniques 1017; 2023-04-01 03:57:11,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-01 03:57:11,633 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/86cb369fced745ed97f7a2bab0e9fc3a 2023-04-01 03:57:11,658 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/750a2a13654e4c05936cb9f67cf6f29f 2023-04-01 03:57:11,663 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/796c947cae1f437990ccb148fc9d43b6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6 2023-04-01 03:57:11,667 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6, entries=100, sequenceid=453, filesize=18.6 K 2023-04-01 03:57:11,668 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/86cb369fced745ed97f7a2bab0e9fc3a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a 2023-04-01 03:57:11,672 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a, entries=100, sequenceid=453, filesize=9.5 K 2023-04-01 03:57:11,673 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/750a2a13654e4c05936cb9f67cf6f29f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f 2023-04-01 03:57:11,677 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f, entries=100, sequenceid=453, filesize=9.5 K 2023-04-01 03:57:11,678 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=127.47 KB/130530 for 57b058ac96577c51eb729e1451adecd4 in 912ms, sequenceid=453, compaction requested=true 2023-04-01 03:57:11,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:11,678 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:11,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:11,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=134.18 KB heapSize=379.55 KB 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:11,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:11,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:11,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141129a7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:11,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:11,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:11,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:11,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:11,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:11,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:11,689 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:11,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:11,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:11,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:11,689 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:11,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,691 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,696 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:11,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:11,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,696 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:11,696 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:11,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:11,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:11,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:11,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321491703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,707 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304010d3c783c0764438b932c09fc7746839a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d3c783c0764438b932c09fc7746839a 2023-04-01 03:57:11,796 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:11,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:11,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:11,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:11,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:11,800 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:11,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:11,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:11,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:11,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,802 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:11,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:11,803 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,803 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:11,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:11,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321491804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:11,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:11,805 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:11,805 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:11,805 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:11,805 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:11,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:11,830 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:11,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b118678[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:11,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:11,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,833 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:11,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:11,834 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:11,834 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,834 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:11,834 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:11,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:11,837 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:11,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:11,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:11,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:11,837 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:11,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,839 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,841 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,842 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,842 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,842 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,843 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:11,843 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:11,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:11,843 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:11,843 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:11,943 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:11,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:11,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:11,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:11,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:11,946 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:11,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:11,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:11,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:11,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,947 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,947 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:11,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:11,948 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,948 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:11,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:11,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:11,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:11,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:11,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:11,950 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:11,950 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:11,950 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:11,950 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:11,950 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:11,951 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:11,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:11,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a40fab2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:11,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,954 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:11,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:11,954 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:11,954 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,954 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:11,957 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:11,959 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:11,959 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:11,959 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:11,959 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:11,959 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:11,961 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,961 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,961 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:11,961 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:11,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:11,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:11,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:11,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:11,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:11,966 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:11,966 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:11,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:11,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321492006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,066 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,069 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,070 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,070 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,073 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,073 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,073 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:12,074 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,074 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,075 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,075 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,075 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,075 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1b6a7f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,078 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,081 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,083 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,083 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,086 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,086 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,086 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,087 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,093 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,093 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,093 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,124 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=696, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/d70e3ab6e5b24b7598fa0fbfe1386bc9 2023-04-01 03:57:12,193 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,196 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,199 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,199 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,199 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:12,199 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,201 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a242a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,204 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,205 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,206 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,207 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,207 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,207 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,209 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:12,211 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,211 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,211 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,311 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321492308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,326 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,327 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,327 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,329 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,329 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,329 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,329 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,329 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,329 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,329 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,329 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:12,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,331 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354f405f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,340 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:12,342 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,342 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,442 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,444 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,444 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:12,448 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1877b92e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,472 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,472 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,474 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,475 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,496 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,497 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,504 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,506 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,506 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/547a0629a3ea4e61822356c1c59cc358 2023-04-01 03:57:12,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f33045865f904820aeb80524ebcb6c11 2023-04-01 03:57:12,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/d70e3ab6e5b24b7598fa0fbfe1386bc9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9 2023-04-01 03:57:12,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9, entries=200, sequenceid=696, filesize=32.3 K 2023-04-01 03:57:12,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/547a0629a3ea4e61822356c1c59cc358 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358 2023-04-01 03:57:12,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358, entries=150, sequenceid=696, filesize=11.9 K 2023-04-01 03:57:12,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/f33045865f904820aeb80524ebcb6c11 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11 2023-04-01 03:57:12,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11, entries=150, sequenceid=696, filesize=11.9 K 2023-04-01 03:57:12,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.50 KB/388608, currentSize=46.96 KB/48090 for 57b058ac96577c51eb729e1451adecd4 in 924ms, sequenceid=696, compaction requested=true 2023-04-01 03:57:12,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:12,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:12,603 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:12,604 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:12,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:12,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:12,606 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:12,606 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:57:12,606 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,606 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/3175d79c2d6b460d97c463e6cff01638, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=45.8 K 2023-04-01 03:57:12,606 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104756 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:12,606 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,606 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:57:12,606 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,606 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/21e068288f884e2f8f1dba1a567cfa9c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=102.3 K 2023-04-01 03:57:12,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,607 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 3175d79c2d6b460d97c463e6cff01638, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427422 2023-04-01 03:57:12,608 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 21e068288f884e2f8f1dba1a567cfa9c, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427422 2023-04-01 03:57:12,608 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 98c3309edc3b473aa99ec69591b3e187, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680321428575 2023-04-01 03:57:12,608 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 645d4e6e0e80400fab6991728f4f085e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680321428575 2023-04-01 03:57:12,608 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 86cb369fced745ed97f7a2bab0e9fc3a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680321430738 2023-04-01 03:57:12,608 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 796c947cae1f437990ccb148fc9d43b6, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680321430738 2023-04-01 03:57:12,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,609 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d70e3ab6e5b24b7598fa0fbfe1386bc9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431634 2023-04-01 03:57:12,609 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 547a0629a3ea4e61822356c1c59cc358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431662 2023-04-01 03:57:12,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,610 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,610 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,612 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,630 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#180 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:12,642 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,642 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,642 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,642 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,642 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:57:12,642 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:12,642 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:12,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:12,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,652 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#181 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:12,663 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/8fd477f57ab040b1a5b1db7ed0c132c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8fd477f57ab040b1a5b1db7ed0c132c0 2023-04-01 03:57:12,667 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into 8fd477f57ab040b1a5b1db7ed0c132c0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:12,667 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:12,667 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=12, startTime=1680321432603; duration=0sec 2023-04-01 03:57:12,667 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:12,667 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:12,668 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:12,668 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:57:12,668 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,668 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/10faae8ca9d144c5b67eb3c1f8d6c8a1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=45.8 K 2023-04-01 03:57:12,668 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 10faae8ca9d144c5b67eb3c1f8d6c8a1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680321427422 2023-04-01 03:57:12,669 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4098f61ebff543d9bea1aaf5b16cea15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680321428575 2023-04-01 03:57:12,669 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 750a2a13654e4c05936cb9f67cf6f29f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680321430738 2023-04-01 03:57:12,669 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f33045865f904820aeb80524ebcb6c11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431662 2023-04-01 03:57:12,676 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401db6b3af03065403b9243ea807bd1a551 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401db6b3af03065403b9243ea807bd1a551 2023-04-01 03:57:12,683 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:12,690 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=706, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5eecc9b61bbf40d4bd43e069aec1a0f1 2023-04-01 03:57:12,703 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/289e162080ca4c2dbf3247f1f33477fc 2023-04-01 03:57:12,724 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d56507697a2545bdae075e56c8ad5629 2023-04-01 03:57:12,731 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5eecc9b61bbf40d4bd43e069aec1a0f1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1 2023-04-01 03:57:12,737 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1, entries=150, sequenceid=706, filesize=25.4 K 2023-04-01 03:57:12,739 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/289e162080ca4c2dbf3247f1f33477fc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc 2023-04-01 03:57:12,743 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc, entries=150, sequenceid=706, filesize=11.9 K 2023-04-01 03:57:12,744 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d56507697a2545bdae075e56c8ad5629 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629 2023-04-01 03:57:12,751 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629, entries=150, sequenceid=706, filesize=11.9 K 2023-04-01 03:57:12,752 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 57b058ac96577c51eb729e1451adecd4 in 110ms, sequenceid=706, compaction requested=false 2023-04-01 03:57:12,752 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:12,753 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,756 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfb70cd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,758 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,763 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,763 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,768 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,768 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,769 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,769 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,769 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:12,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:12,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680321432832, max timestamp=1680321432864Num uniques 859; 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680321432832, max timestamp=1680321432864Num uniques 859; 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680321432832, max timestamp=1680321432864Num uniques 859; 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:57:12,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:57:12,869 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:12,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:12,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:12,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:12,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:12,876 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:12,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:12,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:12,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,877 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,878 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:12,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:12,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:12,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:12,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:12,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:12,884 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,884 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,892 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,892 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321492889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44254 deadline: 1680321492892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:12,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f53740c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:12,895 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:12,895 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321492894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:12,900 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:12,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:12,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:12,904 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:12,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:12,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:12,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:12,904 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:12,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:12,907 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,908 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:12,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:12,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:12,909 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:12,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:12,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:12,910 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,910 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:12,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:12,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:12,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:12,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44222 deadline: 1680321492995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44254 deadline: 1680321492995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:12,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321492998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,010 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,013 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,017 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40519001[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,019 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,019 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,019 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,020 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,021 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,022 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,023 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,024 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,027 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,027 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,027 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,027 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,027 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,027 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,100 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/4dcaead53e5a496dafb49a9a86e9a8b0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4dcaead53e5a496dafb49a9a86e9a8b0 2023-04-01 03:57:13,104 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into 4dcaead53e5a496dafb49a9a86e9a8b0(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:13,104 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:13,104 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=12, startTime=1680321432604; duration=0sec 2023-04-01 03:57:13,104 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:13,127 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,130 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,133 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,133 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,133 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,133 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,134 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,134 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,134 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,134 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,134 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,136 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,137 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,137 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65caffec[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,138 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,138 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,138 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,138 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,140 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,140 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,140 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,141 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,141 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,145 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,149 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,150 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,150 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44254 deadline: 1680321493199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321493199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44222 deadline: 1680321493200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,250 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,253 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018a82b8cc32e84615a9259111e09a5f78 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018a82b8cc32e84615a9259111e09a5f78 2023-04-01 03:57:13,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,254 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,264 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,266 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,266 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,266 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,266 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,279 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62308680[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,287 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,288 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,288 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,288 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,288 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,291 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,291 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,292 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,295 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,295 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,295 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,395 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,398 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,400 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,403 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d76dde[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,407 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,407 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,409 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,410 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,410 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,410 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,410 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,410 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,414 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,416 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,416 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,416 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,479 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/77ecb677e1d74dc88b49e684d9379f39 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/77ecb677e1d74dc88b49e684d9379f39 2023-04-01 03:57:13,484 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into 77ecb677e1d74dc88b49e684d9379f39(size=26.1 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:13,484 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:13,484 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=12, startTime=1680321432603; duration=0sec 2023-04-01 03:57:13,484 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:13,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321493503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:13,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44254 deadline: 1680321493504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44222 deadline: 1680321493505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,516 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,519 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,522 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,522 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,522 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a4a2a9d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,530 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,530 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,536 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,536 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,636 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,639 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,639 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,639 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,639 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,639 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,639 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,648 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,650 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,650 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,650 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,650 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,659 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=718, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5633f23da9b940d2b5f1241ee0ed6cec 2023-04-01 03:57:13,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8cd87a7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,674 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,674 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,677 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,677 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,682 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,686 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,686 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,786 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,789 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,791 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,791 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,793 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,793 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,793 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,793 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2f14b8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,798 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,802 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,802 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,804 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,804 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,804 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,808 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,808 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,908 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:13,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:13,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:13,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:13,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:13,911 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:13,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:13,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:13,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,912 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,912 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,912 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,912 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:13,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:13,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:13,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:13,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:13,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:13,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:13,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:13,913 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,913 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:13,913 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:13,914 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:13,914 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,915 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,915 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:13,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235b3d3e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:13,916 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:13,916 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,917 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:13,917 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:13,917 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:13,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:13,917 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:13,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,918 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,918 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:13,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:13,921 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:13,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:13,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:13,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:13,921 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:13,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:13,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:13,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:13,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:13,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:13,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:13,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:13,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:13,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:13,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:13,927 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:13,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:13,927 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:13,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:13,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:13,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:14,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44254 deadline: 1680321494009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44222 deadline: 1680321494009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 506 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44178 deadline: 1680321494010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,027 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:14,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:14,030 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,038 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,038 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,039 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,039 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:14,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:14,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:14,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:14,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:14,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:14,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:14,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:14,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482ef50a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:14,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:14,056 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8 2023-04-01 03:57:14,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,071 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:14,071 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:14,071 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:14,071 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,071 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:14,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:14,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:14,075 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:14,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:14,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:14,075 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:14,075 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:14,076 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,076 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,076 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,078 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,080 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:14,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,080 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,080 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,180 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:14,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:14,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:14,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:14,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:14,183 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:14,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:14,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:14,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,184 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:14,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:14,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:14,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:14,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:14,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:14,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:14,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:14,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:14,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:14,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:14,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:14,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:14,187 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:14,189 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:14,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c3aa317[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:14,189 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,191 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:14,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:14,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:14,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:14,195 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:14,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:14,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:14,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:14,195 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:14,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,199 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:14,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:14,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:14,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,200 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:14,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,200 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,300 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:14,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:14,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:14,303 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:14,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:14,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:14,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:14,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:14,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:14,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. started... 2023-04-01 03:57:14,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. as already flushing 2023-04-01 03:57:14,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:14,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:14,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aa6f8fc[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:14,308 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:14,308 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,308 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:14,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:14,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:14,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:14,309 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:14,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:14,311 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:14,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:14,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:14,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:14,311 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:14,312 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,312 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,312 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:14,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:14,314 DEBUG [Thread-902] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3659fc7a to 127.0.0.1:50457 2023-04-01 03:57:14,314 DEBUG [Thread-902] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:14,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:14,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:14,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:14,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:14,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:14,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:14,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:14,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:14,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:14,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:14,316 DEBUG [Thread-901] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:14,316 DEBUG [Thread-901] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:14,317 DEBUG [Thread-904] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64b70ca7 to 127.0.0.1:50457 2023-04-01 03:57:14,317 DEBUG [Thread-904] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:14,320 DEBUG [Thread-908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58ee9cda to 127.0.0.1:50457 2023-04-01 03:57:14,320 DEBUG [Thread-908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:14,320 DEBUG [Thread-906] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fde0bbc to 127.0.0.1:50457 2023-04-01 03:57:14,320 DEBUG [Thread-906] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:14,417 DEBUG [Thread-901] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:14,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:14,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d77e1492cf21402d9cbb228044b4a464 2023-04-01 03:57:14,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/5633f23da9b940d2b5f1241ee0ed6cec as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec 2023-04-01 03:57:14,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec, entries=400, sequenceid=718, filesize=59.8 K 2023-04-01 03:57:14,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8 2023-04-01 03:57:14,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8, entries=150, sequenceid=718, filesize=11.9 K 2023-04-01 03:57:14,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/d77e1492cf21402d9cbb228044b4a464 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464 2023-04-01 03:57:14,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464, entries=150, sequenceid=718, filesize=11.9 K 2023-04-01 03:57:14,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 57b058ac96577c51eb729e1451adecd4 in 1700ms, sequenceid=718, compaction requested=true 2023-04-01 03:57:14,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:14,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:14,521 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:14,521 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:14,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:14,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:14,523 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:14,523 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/A is initiating minor compaction (all files) 2023-04-01 03:57:14,523 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/A in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,523 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:14,523 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/77ecb677e1d74dc88b49e684d9379f39, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=111.4 K 2023-04-01 03:57:14,523 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/B is initiating minor compaction (all files) 2023-04-01 03:57:14,523 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/B in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,523 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8fd477f57ab040b1a5b1db7ed0c132c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=36.4 K 2023-04-01 03:57:14,524 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 77ecb677e1d74dc88b49e684d9379f39, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431662 2023-04-01 03:57:14,524 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8fd477f57ab040b1a5b1db7ed0c132c0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431662 2023-04-01 03:57:14,524 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5eecc9b61bbf40d4bd43e069aec1a0f1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1680321431693 2023-04-01 03:57:14,525 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 289e162080ca4c2dbf3247f1f33477fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1680321431693 2023-04-01 03:57:14,525 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5633f23da9b940d2b5f1241ee0ed6cec, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680321432779 2023-04-01 03:57:14,525 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a2ff4cc7e1ea40e8a2b956dfc74d4ba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680321432812 2023-04-01 03:57:14,534 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#B#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:14,537 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#A#compaction#189 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:14,956 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/bb742f635ea54b9b9b46f78910350498 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bb742f635ea54b9b9b46f78910350498 2023-04-01 03:57:14,961 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/B of 57b058ac96577c51eb729e1451adecd4 into bb742f635ea54b9b9b46f78910350498(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:14,961 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:14,961 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/B, priority=13, startTime=1680321434521; duration=0sec 2023-04-01 03:57:14,961 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:14,961 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:14,962 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:14,962 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 57b058ac96577c51eb729e1451adecd4/C is initiating minor compaction (all files) 2023-04-01 03:57:14,962 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 57b058ac96577c51eb729e1451adecd4/C in TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:14,962 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4dcaead53e5a496dafb49a9a86e9a8b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp, totalSize=36.4 K 2023-04-01 03:57:14,963 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4dcaead53e5a496dafb49a9a86e9a8b0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680321431662 2023-04-01 03:57:14,963 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d56507697a2545bdae075e56c8ad5629, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1680321431693 2023-04-01 03:57:14,963 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d77e1492cf21402d9cbb228044b4a464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680321432812 2023-04-01 03:57:14,969 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 57b058ac96577c51eb729e1451adecd4#C#compaction#191 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:15,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:15,018 DEBUG [Thread-895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ec2774b to 127.0.0.1:50457 2023-04-01 03:57:15,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-01 03:57:15,018 DEBUG [Thread-895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,018 DEBUG [Thread-899] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x710671dd to 127.0.0.1:50457 2023-04-01 03:57:15,018 DEBUG [Thread-899] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:15,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:15,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:15,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:15,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:15,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:15,021 DEBUG [Thread-897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e0f4d47 to 127.0.0.1:50457 2023-04-01 03:57:15,021 DEBUG [Thread-897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,132 DEBUG [Thread-893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63f0ce11 to 127.0.0.1:50457 2023-04-01 03:57:15,132 DEBUG [Thread-893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,249 DEBUG [Thread-891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54ec4d71 to 127.0.0.1:50457 2023-04-01 03:57:15,249 DEBUG [Thread-891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,249 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:57:15,249 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-01 03:57:15,249 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-01 03:57:15,249 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 399 2023-04-01 03:57:15,249 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5112 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5190 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 2199 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 6597 rows 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 2205 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 6615 rows 2023-04-01 03:57:15,250 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:57:15,250 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6902ce3c to 127.0.0.1:50457 2023-04-01 03:57:15,250 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:15,251 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:57:15,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:57:15,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:15,259 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321435259"}]},"ts":"1680321435259"} 2023-04-01 03:57:15,260 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:57:15,261 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:57:15,262 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, UNASSIGN}] 2023-04-01 03:57:15,262 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, UNASSIGN 2023-04-01 03:57:15,263 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:15,263 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321435263"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321435263"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321435263"}]},"ts":"1680321435263"} 2023-04-01 03:57:15,264 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:15,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:15,377 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/9cfa20ca60ca4328a65880c4b5c11ea6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9cfa20ca60ca4328a65880c4b5c11ea6 2023-04-01 03:57:15,382 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/C of 57b058ac96577c51eb729e1451adecd4 into 9cfa20ca60ca4328a65880c4b5c11ea6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:15,382 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:15,382 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/C, priority=13, startTime=1680321434521; duration=0sec 2023-04-01 03:57:15,382 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:15,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:15,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 57b058ac96577c51eb729e1451adecd4, disabling compactions & flushes 2023-04-01 03:57:15,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:15,431 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012cd540da89d04aceaa13667c9eec7a65 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cd540da89d04aceaa13667c9eec7a65 2023-04-01 03:57:15,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:15,769 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/7a034a0d02cc4905be7598133a5bb534 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/7a034a0d02cc4905be7598133a5bb534 2023-04-01 03:57:15,778 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 57b058ac96577c51eb729e1451adecd4/A of 57b058ac96577c51eb729e1451adecd4 into 7a034a0d02cc4905be7598133a5bb534(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:57:15,778 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:15,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:15,778 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4., storeName=57b058ac96577c51eb729e1451adecd4/A, priority=13, startTime=1680321434520; duration=1sec 2023-04-01 03:57:15,778 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:15,837 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=757, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/295fb92a8e994f46b4dbd76af98eb0fd 2023-04-01 03:57:15,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:16,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/4d809a2f8dcf4274b774a0ba4510c7ae 2023-04-01 03:57:16,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:16,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/9e96af7af094457788890ba00b517613 2023-04-01 03:57:16,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/295fb92a8e994f46b4dbd76af98eb0fd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/295fb92a8e994f46b4dbd76af98eb0fd 2023-04-01 03:57:16,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/295fb92a8e994f46b4dbd76af98eb0fd, entries=150, sequenceid=757, filesize=25.4 K 2023-04-01 03:57:16,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/4d809a2f8dcf4274b774a0ba4510c7ae as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/4d809a2f8dcf4274b774a0ba4510c7ae 2023-04-01 03:57:16,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/4d809a2f8dcf4274b774a0ba4510c7ae, entries=150, sequenceid=757, filesize=11.9 K 2023-04-01 03:57:16,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/9e96af7af094457788890ba00b517613 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9e96af7af094457788890ba00b517613 2023-04-01 03:57:16,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9e96af7af094457788890ba00b517613, entries=150, sequenceid=757, filesize=11.9 K 2023-04-01 03:57:16,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=20.13 KB/20610 for 57b058ac96577c51eb729e1451adecd4 in 1675ms, sequenceid=757, compaction requested=false 2023-04-01 03:57:16,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. after waiting 0 ms 2023-04-01 03:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 57b058ac96577c51eb729e1451adecd4 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=A 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=B 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 57b058ac96577c51eb729e1451adecd4, store=C 2023-04-01 03:57:16,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:17,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040181cc3f04f290445abe32ab8c9fb9bb27 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181cc3f04f290445abe32ab8c9fb9bb27 2023-04-01 03:57:17,288 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-01 03:57:17,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:17,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=765, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/0b90e2086f4f44fea6e106487bba691a 2023-04-01 03:57:17,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/e8c0443c94694a0fa7f787e03144b4f1 2023-04-01 03:57:18,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/5d6bc7b16ddb4d0ea227d675727b6f09 2023-04-01 03:57:18,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/A/0b90e2086f4f44fea6e106487bba691a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/0b90e2086f4f44fea6e106487bba691a 2023-04-01 03:57:18,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/0b90e2086f4f44fea6e106487bba691a, entries=100, sequenceid=765, filesize=18.6 K 2023-04-01 03:57:18,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/B/e8c0443c94694a0fa7f787e03144b4f1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/e8c0443c94694a0fa7f787e03144b4f1 2023-04-01 03:57:18,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/e8c0443c94694a0fa7f787e03144b4f1, entries=100, sequenceid=765, filesize=9.5 K 2023-04-01 03:57:18,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/.tmp/C/5d6bc7b16ddb4d0ea227d675727b6f09 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/5d6bc7b16ddb4d0ea227d675727b6f09 2023-04-01 03:57:18,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/5d6bc7b16ddb4d0ea227d675727b6f09, entries=100, sequenceid=765, filesize=9.5 K 2023-04-01 03:57:18,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 57b058ac96577c51eb729e1451adecd4 in 1727ms, sequenceid=765, compaction requested=true 2023-04-01 03:57:18,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/988ff6fd165941588527f1ebc57a5ed0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d0a0718efe874635b97510f4204f0101, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/a8d6ec8091584273b7a75c3b1740b191, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/21e068288f884e2f8f1dba1a567cfa9c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/77ecb677e1d74dc88b49e684d9379f39, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec] to archive 2023-04-01 03:57:18,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:18,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/3f726dcedf064fa9a35faa7e30c0eabd 2023-04-01 03:57:18,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/1f1d985e8a8c4b878b80124e67be35b6 2023-04-01 03:57:18,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/2946ddca31da4c37810697438d4130e3 2023-04-01 03:57:18,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/058c74d4c0b940469c20672e62035e37 2023-04-01 03:57:18,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/988ff6fd165941588527f1ebc57a5ed0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/988ff6fd165941588527f1ebc57a5ed0 2023-04-01 03:57:18,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/781dec769d044c3483aea749551c6bc4 2023-04-01 03:57:18,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/c01687baff4e4ebd83e4f614037fcc3a 2023-04-01 03:57:18,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d0a0718efe874635b97510f4204f0101 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d0a0718efe874635b97510f4204f0101 2023-04-01 03:57:18,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/fee8398b21a14befa01b439a4cdcaab3 2023-04-01 03:57:18,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/22facf1421b845ae980a51b62caf91cd 2023-04-01 03:57:18,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5074e1ccd8b844aea6adb88b896ed6b7 2023-04-01 03:57:18,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/8af57e902c6c4b8caf1fec518a37f211 2023-04-01 03:57:18,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/29d2a36a68fa4677a08287ecb8cc07b9 2023-04-01 03:57:18,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/f5134dacdefe48069022612508d4608e 2023-04-01 03:57:18,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/a8d6ec8091584273b7a75c3b1740b191 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/a8d6ec8091584273b7a75c3b1740b191 2023-04-01 03:57:18,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/4a1d09abef644fa3958520c869aef266 2023-04-01 03:57:18,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/e7701ca422ab4b31a163bc677528d80a 2023-04-01 03:57:18,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/21e068288f884e2f8f1dba1a567cfa9c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/21e068288f884e2f8f1dba1a567cfa9c 2023-04-01 03:57:18,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/645d4e6e0e80400fab6991728f4f085e 2023-04-01 03:57:18,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/796c947cae1f437990ccb148fc9d43b6 2023-04-01 03:57:18,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/d70e3ab6e5b24b7598fa0fbfe1386bc9 2023-04-01 03:57:18,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/77ecb677e1d74dc88b49e684d9379f39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/77ecb677e1d74dc88b49e684d9379f39 2023-04-01 03:57:18,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5eecc9b61bbf40d4bd43e069aec1a0f1 2023-04-01 03:57:18,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/5633f23da9b940d2b5f1241ee0ed6cec 2023-04-01 03:57:18,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/01da5847b07746afa5743a7864d79834, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/b023553c19f54f74925426c9f25587ad, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c1e6cd87a9b64fd6ab1b274ea8c28b03, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/3175d79c2d6b460d97c463e6cff01638, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8fd477f57ab040b1a5b1db7ed0c132c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8] to archive 2023-04-01 03:57:18,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:18,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a41455a4cedd407890a66553b6e40620 2023-04-01 03:57:18,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/ad3e8fb4145d44b680558be3c5492064 2023-04-01 03:57:18,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c83d9df9f6d64c79b3b2e69519c04e47 2023-04-01 03:57:18,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/01da5847b07746afa5743a7864d79834 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/01da5847b07746afa5743a7864d79834 2023-04-01 03:57:18,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/80c79ab8438d406289ff7747029ca4ae 2023-04-01 03:57:18,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bcc66a57ba674a29bbc44222030269a7 2023-04-01 03:57:18,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/b023553c19f54f74925426c9f25587ad to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/b023553c19f54f74925426c9f25587ad 2023-04-01 03:57:18,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/31a6cd4c6ff6485ca799d2c9d5e9c1f6 2023-04-01 03:57:18,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/869fcc1b83f844228c56523566417bef 2023-04-01 03:57:18,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/eb0b422ba22e4d248f141b8eaf6be80b 2023-04-01 03:57:18,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/1f51172194d445adaf6784ec6080d81e 2023-04-01 03:57:18,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8a5296d27dfe48c8abcc79c63932f1c0 2023-04-01 03:57:18,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/68400618ab74462a8e815a06579c8a38 2023-04-01 03:57:18,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c1e6cd87a9b64fd6ab1b274ea8c28b03 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/c1e6cd87a9b64fd6ab1b274ea8c28b03 2023-04-01 03:57:18,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/9be14d2065684a41b01f24a6bc97d3b2 2023-04-01 03:57:18,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7c1586b504b14da59fbf80c4c7f1b7a6 2023-04-01 03:57:18,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/3175d79c2d6b460d97c463e6cff01638 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/3175d79c2d6b460d97c463e6cff01638 2023-04-01 03:57:18,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/7fe2438c9dfd42a1a7eeb5610de023c2 2023-04-01 03:57:18,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/98c3309edc3b473aa99ec69591b3e187 2023-04-01 03:57:18,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/86cb369fced745ed97f7a2bab0e9fc3a 2023-04-01 03:57:18,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8fd477f57ab040b1a5b1db7ed0c132c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/8fd477f57ab040b1a5b1db7ed0c132c0 2023-04-01 03:57:18,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/547a0629a3ea4e61822356c1c59cc358 2023-04-01 03:57:18,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/289e162080ca4c2dbf3247f1f33477fc 2023-04-01 03:57:18,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/a2ff4cc7e1ea40e8a2b956dfc74d4ba8 2023-04-01 03:57:18,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4129e79d25124a2aafc81220393d6628, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/a82f18fa97194925b5e5d045f3ddded6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/59d51c9034524200ba12d46881071d3c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/10faae8ca9d144c5b67eb3c1f8d6c8a1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4dcaead53e5a496dafb49a9a86e9a8b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464] to archive 2023-04-01 03:57:18,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:18,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/192a34e0d24c46558633b79df86b2dd0 2023-04-01 03:57:18,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/aeb275038d9b4a36b6b854772e5ebdd1 2023-04-01 03:57:18,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/1c3ec9037c1b444cb31e9c98dcbeabb4 2023-04-01 03:57:18,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4129e79d25124a2aafc81220393d6628 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4129e79d25124a2aafc81220393d6628 2023-04-01 03:57:18,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/e54aa5e4573d4c9487b762a0507677d6 2023-04-01 03:57:18,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/921879f98c1c404ca4a7016382fa9a83 2023-04-01 03:57:18,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/a82f18fa97194925b5e5d045f3ddded6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/a82f18fa97194925b5e5d045f3ddded6 2023-04-01 03:57:18,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/34e488195e3846d1bf513271ba1a289e 2023-04-01 03:57:18,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/829127184ad840bcaf4fd2534bf98750 2023-04-01 03:57:18,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/50cc650fffa54be3818d9592a63383a4 2023-04-01 03:57:18,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f7193abaa13d4f969cae439406daecc5 2023-04-01 03:57:18,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/6f2f6c7d439e473a8290d60d8e889793 2023-04-01 03:57:18,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d1b3d01c037349dbb76ec241f8bef394 2023-04-01 03:57:18,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/59d51c9034524200ba12d46881071d3c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/59d51c9034524200ba12d46881071d3c 2023-04-01 03:57:18,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f4bdd149309a47918985c09007d4fe1e 2023-04-01 03:57:18,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/adc6fb6f1ced410ea22ec0b4d4aea9ea 2023-04-01 03:57:18,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/10faae8ca9d144c5b67eb3c1f8d6c8a1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/10faae8ca9d144c5b67eb3c1f8d6c8a1 2023-04-01 03:57:18,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9dfff0049b894c70a1b8d6fc282d421f 2023-04-01 03:57:18,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4098f61ebff543d9bea1aaf5b16cea15 2023-04-01 03:57:18,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/750a2a13654e4c05936cb9f67cf6f29f 2023-04-01 03:57:18,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4dcaead53e5a496dafb49a9a86e9a8b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/4dcaead53e5a496dafb49a9a86e9a8b0 2023-04-01 03:57:18,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/f33045865f904820aeb80524ebcb6c11 2023-04-01 03:57:18,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d56507697a2545bdae075e56c8ad5629 2023-04-01 03:57:18,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/d77e1492cf21402d9cbb228044b4a464 2023-04-01 03:57:18,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits/768.seqid, newMaxSeqId=768, maxSeqId=4 2023-04-01 03:57:18,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4. 2023-04-01 03:57:18,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 57b058ac96577c51eb729e1451adecd4: 2023-04-01 03:57:18,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:18,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=57b058ac96577c51eb729e1451adecd4, regionState=CLOSED 2023-04-01 03:57:18,633 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321438633"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321438633"}]},"ts":"1680321438633"} 2023-04-01 03:57:18,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-01 03:57:18,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 57b058ac96577c51eb729e1451adecd4, server=jenkins-hbase20.apache.org,36251,1680321375816 in 3.3700 sec 2023-04-01 03:57:18,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-01 03:57:18,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=57b058ac96577c51eb729e1451adecd4, UNASSIGN in 3.3740 sec 2023-04-01 03:57:18,638 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321438637"}]},"ts":"1680321438637"} 2023-04-01 03:57:18,638 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:57:18,640 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:57:18,643 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3900 sec 2023-04-01 03:57:19,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 03:57:19,365 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-01 03:57:19,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:57:19,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,369 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,370 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-01 03:57:19,376 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:19,383 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits] 2023-04-01 03:57:19,386 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/0b90e2086f4f44fea6e106487bba691a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/0b90e2086f4f44fea6e106487bba691a 2023-04-01 03:57:19,397 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/295fb92a8e994f46b4dbd76af98eb0fd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/295fb92a8e994f46b4dbd76af98eb0fd 2023-04-01 03:57:19,399 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/7a034a0d02cc4905be7598133a5bb534 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/A/7a034a0d02cc4905be7598133a5bb534 2023-04-01 03:57:19,401 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/4d809a2f8dcf4274b774a0ba4510c7ae to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/4d809a2f8dcf4274b774a0ba4510c7ae 2023-04-01 03:57:19,402 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bb742f635ea54b9b9b46f78910350498 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/bb742f635ea54b9b9b46f78910350498 2023-04-01 03:57:19,403 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/e8c0443c94694a0fa7f787e03144b4f1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/B/e8c0443c94694a0fa7f787e03144b4f1 2023-04-01 03:57:19,405 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/5d6bc7b16ddb4d0ea227d675727b6f09 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/5d6bc7b16ddb4d0ea227d675727b6f09 2023-04-01 03:57:19,406 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9cfa20ca60ca4328a65880c4b5c11ea6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9cfa20ca60ca4328a65880c4b5c11ea6 2023-04-01 03:57:19,407 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9e96af7af094457788890ba00b517613 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/C/9e96af7af094457788890ba00b517613 2023-04-01 03:57:19,410 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits/768.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4/recovered.edits/768.seqid 2023-04-01 03:57:19,411 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/57b058ac96577c51eb729e1451adecd4 2023-04-01 03:57:19,411 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:57:19,411 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:57:19,415 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 03:57:19,429 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d3c783c0764438b932c09fc7746839a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d3c783c0764438b932c09fc7746839a 2023-04-01 03:57:19,430 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120ed1064686e4954a250eca5d6582fe8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120ed1064686e4954a250eca5d6582fe8 2023-04-01 03:57:19,431 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040124db7abe002e4173908230027fc287bb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040124db7abe002e4173908230027fc287bb 2023-04-01 03:57:19,432 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012be7459f2f384c889c27f5ec4a372db1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012be7459f2f384c889c27f5ec4a372db1 2023-04-01 03:57:19,433 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cd540da89d04aceaa13667c9eec7a65 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cd540da89d04aceaa13667c9eec7a65 2023-04-01 03:57:19,434 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014e587178c94f4f6d8e950c896f8edf3e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014e587178c94f4f6d8e950c896f8edf3e 2023-04-01 03:57:19,444 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015ce93681aedf41d4ae4f2ec4fe84c458 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015ce93681aedf41d4ae4f2ec4fe84c458 2023-04-01 03:57:19,445 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015f03172b850c41ed9218facf2207fd64 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015f03172b850c41ed9218facf2207fd64 2023-04-01 03:57:19,448 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040164ee24b309564cfb84f26f4abd64450e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040164ee24b309564cfb84f26f4abd64450e 2023-04-01 03:57:19,452 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017f58e14d683f446c9b2123873bac96a1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017f58e14d683f446c9b2123873bac96a1 2023-04-01 03:57:19,457 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181cc3f04f290445abe32ab8c9fb9bb27 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181cc3f04f290445abe32ab8c9fb9bb27 2023-04-01 03:57:19,459 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040183a30e2d61c5416ea573bfaecfa627c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040183a30e2d61c5416ea573bfaecfa627c0 2023-04-01 03:57:19,465 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018a82b8cc32e84615a9259111e09a5f78 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018a82b8cc32e84615a9259111e09a5f78 2023-04-01 03:57:19,467 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019684dfa5fa4149fa8bcf2c26bf35a9fc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019684dfa5fa4149fa8bcf2c26bf35a9fc 2023-04-01 03:57:19,469 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040199f25b0e7d2d483594280657e4dae3c9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040199f25b0e7d2d483594280657e4dae3c9 2023-04-01 03:57:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-01 03:57:19,474 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e4b262c59234462b263167b5008d621 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e4b262c59234462b263167b5008d621 2023-04-01 03:57:19,475 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c06565420bfd4c198f033fad664c6810 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c06565420bfd4c198f033fad664c6810 2023-04-01 03:57:19,477 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c4f6ef50e21b4235832c36d274013ccf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c4f6ef50e21b4235832c36d274013ccf 2023-04-01 03:57:19,480 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cccf0de61f9f41f3ad5626f7989e16b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cccf0de61f9f41f3ad5626f7989e16b0 2023-04-01 03:57:19,482 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ccf67591dcf44fe698d6fa4fbaa3329f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ccf67591dcf44fe698d6fa4fbaa3329f 2023-04-01 03:57:19,483 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401db6b3af03065403b9243ea807bd1a551 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401db6b3af03065403b9243ea807bd1a551 2023-04-01 03:57:19,483 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:57:19,485 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,487 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:57:19,489 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:57:19,490 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,490 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:57:19,491 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321439490"}]},"ts":"9223372036854775807"} 2023-04-01 03:57:19,492 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:57:19,492 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 57b058ac96577c51eb729e1451adecd4, NAME => 'TestAcidGuarantees,,1680321411049.57b058ac96577c51eb729e1451adecd4.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:57:19,492 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:57:19,492 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321439492"}]},"ts":"9223372036854775807"} 2023-04-01 03:57:19,494 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:57:19,495 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,497 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 129 msec 2023-04-01 03:57:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-01 03:57:19,672 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-01 03:57:19,686 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229 (was 229), OpenFileDescriptor=553 (was 560), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=364 (was 265) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=7809 (was 7999) 2023-04-01 03:57:19,698 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=364, ProcessCount=167, AvailableMemoryMB=7808 2023-04-01 03:57:19,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:57:19,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:57:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:19,710 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:57:19,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-01 03:57:19,710 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:57:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:19,714 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:19,715 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 empty. 2023-04-01 03:57:19,716 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:19,716 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:57:19,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:20,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:20,130 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:57:20,133 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 152d4d46bcbd27b89955640499bc0d72, NAME => 'TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:57:20,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 152d4d46bcbd27b89955640499bc0d72, disabling compactions & flushes 2023-04-01 03:57:20,542 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. after waiting 0 ms 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,542 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,542 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:20,546 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:57:20,546 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321440546"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321440546"}]},"ts":"1680321440546"} 2023-04-01 03:57:20,547 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:57:20,550 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:57:20,550 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321440550"}]},"ts":"1680321440550"} 2023-04-01 03:57:20,551 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:57:20,555 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, ASSIGN}] 2023-04-01 03:57:20,557 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, ASSIGN 2023-04-01 03:57:20,558 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:57:20,709 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=152d4d46bcbd27b89955640499bc0d72, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:20,710 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321440709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321440709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321440709"}]},"ts":"1680321440709"} 2023-04-01 03:57:20,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:20,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:20,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 152d4d46bcbd27b89955640499bc0d72, NAME => 'TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:57:20,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:57:20,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,870 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,875 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A 2023-04-01 03:57:20,875 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A 2023-04-01 03:57:20,875 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:20,875 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 152d4d46bcbd27b89955640499bc0d72 columnFamilyName A 2023-04-01 03:57:20,878 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(310): Store=152d4d46bcbd27b89955640499bc0d72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:20,878 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,883 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B 2023-04-01 03:57:20,883 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B 2023-04-01 03:57:20,883 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:20,884 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 152d4d46bcbd27b89955640499bc0d72 columnFamilyName B 2023-04-01 03:57:20,884 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(310): Store=152d4d46bcbd27b89955640499bc0d72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:20,884 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,888 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C 2023-04-01 03:57:20,888 DEBUG [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C 2023-04-01 03:57:20,888 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:20,889 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 152d4d46bcbd27b89955640499bc0d72 columnFamilyName C 2023-04-01 03:57:20,889 INFO [StoreOpener-152d4d46bcbd27b89955640499bc0d72-1] regionserver.HStore(310): Store=152d4d46bcbd27b89955640499bc0d72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:20,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:57:20,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:20,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:57:20,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 152d4d46bcbd27b89955640499bc0d72; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62898975, jitterRate=-0.06273223459720612}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:57:20,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:20,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., pid=30, masterSystemTime=1680321440865 2023-04-01 03:57:20,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:20,900 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=152d4d46bcbd27b89955640499bc0d72, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:20,901 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321440900"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321440900"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321440900"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321440900"}]},"ts":"1680321440900"} 2023-04-01 03:57:20,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-01 03:57:20,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 in 188 msec 2023-04-01 03:57:20,904 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-01 03:57:20,904 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, ASSIGN in 347 msec 2023-04-01 03:57:20,904 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:57:20,905 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321440904"}]},"ts":"1680321440904"} 2023-04-01 03:57:20,905 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:57:20,914 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:57:20,915 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2110 sec 2023-04-01 03:57:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 03:57:21,816 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-01 03:57:21,817 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x10f967f5 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,834 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@297bb06a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,840 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,841 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:21,846 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:57:21,851 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:57:21,852 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x72d06f87 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,861 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29cd68e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,862 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x553863db to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,881 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@246592ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,882 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3090fec3 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,892 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262075b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,892 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x229d4b93 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,911 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15ccdfe5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,912 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x5dc8411c to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,916 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@154148e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,917 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6cc75c6c to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,927 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@735e3771, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,927 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x2bf0999a to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,938 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d8294b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,938 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x5f634c21 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,949 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7070321e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,949 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x488456ec to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,964 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dc31999, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,964 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x14784858 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:21,970 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e215f0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:21,983 DEBUG [hconnection-0x25aea81c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:21,991 DEBUG [hconnection-0x4c93fd56-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,992 DEBUG [hconnection-0x6e22bfec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,992 DEBUG [hconnection-0x17d55cb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,994 DEBUG [hconnection-0x1b1cb91d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,994 DEBUG [hconnection-0x43ad8e70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,994 DEBUG [hconnection-0x6ae5ec24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:21,995 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,000 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,000 DEBUG [hconnection-0x68d71c5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:22,000 DEBUG [hconnection-0x42fd44cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:22,002 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,002 DEBUG [hconnection-0x3dc7784b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:22,002 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,002 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,003 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,003 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:22,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,017 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,017 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,017 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,020 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:22,021 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:22,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:57:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e805b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,042 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,042 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:22,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,060 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,060 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,063 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,063 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,063 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:22,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:22,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2e364b5662334f46a717f8e65662da31 2023-04-01 03:57:22,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,074 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:22,074 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:22,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:22,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/26cf416f9c604b359d647ae0d82dfa43 2023-04-01 03:57:22,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/221aeb5d99ac4ee6a62471b695ab69b0 2023-04-01 03:57:22,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2e364b5662334f46a717f8e65662da31 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31 2023-04-01 03:57:22,175 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:22,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:22,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:22,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31, entries=150, sequenceid=14, filesize=11.6 K 2023-04-01 03:57:22,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,180 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,180 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,180 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,180 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,185 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/26cf416f9c604b359d647ae0d82dfa43 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43 2023-04-01 03:57:22,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,205 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,205 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,205 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,205 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,205 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:22,206 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,210 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,210 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f57abf9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,211 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,211 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,212 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,213 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,213 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43, entries=150, sequenceid=14, filesize=11.6 K 2023-04-01 03:57:22,214 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,215 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,215 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,215 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/221aeb5d99ac4ee6a62471b695ab69b0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0 2023-04-01 03:57:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:22,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,227 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,227 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,228 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,228 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0, entries=150, sequenceid=14, filesize=11.6 K 2023-04-01 03:57:22,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 152d4d46bcbd27b89955640499bc0d72 in 214ms, sequenceid=14, compaction requested=false 2023-04-01 03:57:22,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,262 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:22,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:22,268 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:22,268 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:22,368 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:22,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:22,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:22,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,381 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,385 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,386 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,386 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,388 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,388 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,388 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:57:22,388 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:22,388 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,389 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:22,389 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,389 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:22,389 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:22,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:22,416 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/5ab17cc8eb5644b1b018a5f9dcff5f67 2023-04-01 03:57:22,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,453 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/0ea0182f57a740d49fd12dae94e2d28e 2023-04-01 03:57:22,484 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/ccdf941b18114ba7b554c3eca08e87d9 2023-04-01 03:57:22,488 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/5ab17cc8eb5644b1b018a5f9dcff5f67 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67 2023-04-01 03:57:22,498 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67, entries=150, sequenceid=34, filesize=11.6 K 2023-04-01 03:57:22,499 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/0ea0182f57a740d49fd12dae94e2d28e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e 2023-04-01 03:57:22,511 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e, entries=150, sequenceid=34, filesize=11.6 K 2023-04-01 03:57:22,512 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/ccdf941b18114ba7b554c3eca08e87d9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9 2023-04-01 03:57:22,518 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9, entries=150, sequenceid=34, filesize=11.6 K 2023-04-01 03:57:22,519 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 152d4d46bcbd27b89955640499bc0d72 in 131ms, sequenceid=34, compaction requested=false 2023-04-01 03:57:22,519 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:22,519 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,520 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,520 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,520 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c8d1f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:22,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,534 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,534 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:22,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:22,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:22,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:22,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:22,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:22,548 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:22,548 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:22,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,648 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:22,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,658 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,658 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,659 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:22,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,663 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be0f42a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,665 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,665 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,666 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,667 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,670 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,670 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,672 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,672 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,675 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:22,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:22,675 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:22,675 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:22,675 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:22,777 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:22,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,781 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,785 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,787 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,794 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,795 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,795 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,796 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,796 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:22,796 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,796 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,796 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,796 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,796 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,796 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,798 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb8c155[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,803 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,816 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,816 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,818 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,819 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:22,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,821 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:22,821 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,821 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:22,821 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:22,823 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,823 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321502868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321502869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321502870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321502871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321502871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,921 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:22,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:22,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:22,925 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:22,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:22,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:22,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,926 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:22,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:22,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:22,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:22,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:22,927 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,927 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:22,928 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:22,928 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:22,932 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df2ad38[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,934 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:22,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:22,952 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:22,952 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:22,952 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:22,954 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:22,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:22,957 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:22,957 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:22,957 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:22,957 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:22,957 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:22,960 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,960 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:22,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:22,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:22,974 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/15bba15550be409cb408a5c3a3f9a628 2023-04-01 03:57:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,985 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,004 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,005 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,005 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,105 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:23,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:23,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:23,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:23,116 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:23,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:23,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:23,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,121 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,121 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,122 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,122 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:23,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:23,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:23,123 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,123 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:23,123 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:23,123 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:23,123 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:23,124 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,124 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,124 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:23,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7433ec[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:23,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:23,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,144 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:23,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:23,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:23,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:23,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:23,166 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:23,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:23,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:23,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:23,166 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:23,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321503172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321503174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321503174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321503176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321503176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,188 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,189 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,189 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,189 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,289 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:23,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:23,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:23,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:23,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:23,298 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:23,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:23,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:23,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,299 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,301 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:23,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:23,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:23,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:23,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:23,303 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:23,304 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,304 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:23,304 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:23,304 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:23,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:23,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1339d58c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:23,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:23,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,337 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:23,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:23,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,338 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:23,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:23,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:23,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:23,348 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:23,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:23,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:23,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:23,348 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:23,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,365 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,365 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,366 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,366 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,366 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/5d7bb1e32b1a4c09ac17511f3be9a82e 2023-04-01 03:57:23,466 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:23,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:23,473 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:23,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:23,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:23,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:23,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:23,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:23,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:23,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,487 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:23,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a720cd7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:23,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:23,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:23,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:23,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:23,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:23,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,498 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:23,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:23,501 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:23,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:23,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:23,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:23,501 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:23,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,504 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,519 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,519 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,520 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,520 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,620 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:23,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:23,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:23,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:23,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:23,626 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:23,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:23,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:23,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,629 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,630 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,630 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:23,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:23,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:23,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:23,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:23,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:23,631 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:23,635 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,635 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:23,636 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:23,636 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:23,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:23,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:23,636 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:23,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,655 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,655 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:23,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201e7cc[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:23,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:23,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,662 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:23,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:23,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:23,671 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:23,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321503679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321503680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321503683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321503683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:23,701 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321503690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,701 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:23,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:23,711 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:23,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:23,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:23,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:23,711 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:23,712 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,712 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,712 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,713 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,721 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,721 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,721 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,721 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,821 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:23,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:23,831 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:23,831 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,831 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:23,831 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:23,831 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,831 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,833 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:23,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:23,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:23,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/7f1f53982f1d424ab041fcb951022772 2023-04-01 03:57:23,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:23,837 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,837 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:23,837 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:23,837 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:23,837 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:23,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,839 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,839 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:23,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c545f33[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:23,852 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:23,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,853 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,853 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:23,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:23,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:23,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:23,854 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:23,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:23,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:23,858 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:23,858 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:23,858 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:23,858 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:23,858 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:23,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/15bba15550be409cb408a5c3a3f9a628 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628 2023-04-01 03:57:23,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628, entries=150, sequenceid=51, filesize=11.6 K 2023-04-01 03:57:23,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/5d7bb1e32b1a4c09ac17511f3be9a82e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e 2023-04-01 03:57:23,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e, entries=150, sequenceid=51, filesize=11.6 K 2023-04-01 03:57:23,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/7f1f53982f1d424ab041fcb951022772 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772 2023-04-01 03:57:23,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:23,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772, entries=150, sequenceid=51, filesize=11.6 K 2023-04-01 03:57:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:23,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 1352ms, sequenceid=51, compaction requested=true 2023-04-01 03:57:23,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:23,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:23,887 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:23,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:23,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:23,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,887 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,890 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:23,891 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/B is initiating minor compaction (all files) 2023-04-01 03:57:23,891 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/B in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,891 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.8 K 2023-04-01 03:57:23,891 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:23,891 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 26cf416f9c604b359d647ae0d82dfa43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680321442017 2023-04-01 03:57:23,891 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/A is initiating minor compaction (all files) 2023-04-01 03:57:23,891 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/A in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,891 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.8 K 2023-04-01 03:57:23,892 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0ea0182f57a740d49fd12dae94e2d28e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680321442076 2023-04-01 03:57:23,892 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2e364b5662334f46a717f8e65662da31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680321442017 2023-04-01 03:57:23,892 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 5d7bb1e32b1a4c09ac17511f3be9a82e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:23,892 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5ab17cc8eb5644b1b018a5f9dcff5f67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680321442076 2023-04-01 03:57:23,892 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 15bba15550be409cb408a5c3a3f9a628, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:23,899 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:23,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:23,902 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#A#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:23,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:23,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:23,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,905 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:23,906 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:23,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:23,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:23,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:23,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:23,960 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/b8ad8a48d8e945a296157671f18a4406 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b8ad8a48d8e945a296157671f18a4406 2023-04-01 03:57:23,973 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/A of 152d4d46bcbd27b89955640499bc0d72 into b8ad8a48d8e945a296157671f18a4406(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:23,973 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:23,973 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/A, priority=13, startTime=1680321443887; duration=0sec 2023-04-01 03:57:23,973 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:23,974 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:23,975 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:23,975 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/C is initiating minor compaction (all files) 2023-04-01 03:57:23,975 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/C in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:23,976 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.8 K 2023-04-01 03:57:23,976 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 221aeb5d99ac4ee6a62471b695ab69b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680321442017 2023-04-01 03:57:23,976 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ccdf941b18114ba7b554c3eca08e87d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680321442076 2023-04-01 03:57:23,976 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7f1f53982f1d424ab041fcb951022772, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:23,986 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#C#compaction#209 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:24,006 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:24,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:24,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:24,009 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:24,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:24,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:24,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:24,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:24,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:24,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:24,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:24,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:24,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:24,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:24,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:24,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:24,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:24,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:24,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:24,014 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:24,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:24,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:24,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:24,198 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:57:24,356 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/58b7139775e448918deaa87a50f57cc3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/58b7139775e448918deaa87a50f57cc3 2023-04-01 03:57:24,391 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/B of 152d4d46bcbd27b89955640499bc0d72 into 58b7139775e448918deaa87a50f57cc3(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:24,391 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:24,391 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/B, priority=13, startTime=1680321443887; duration=0sec 2023-04-01 03:57:24,391 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:24,419 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/9390bc8038284059ae9eb4fe1753d8c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9390bc8038284059ae9eb4fe1753d8c0 2023-04-01 03:57:24,431 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/C of 152d4d46bcbd27b89955640499bc0d72 into 9390bc8038284059ae9eb4fe1753d8c0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:24,431 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:24,431 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/C, priority=13, startTime=1680321443887; duration=0sec 2023-04-01 03:57:24,431 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:24,440 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2a20bd6e3c994cedbc92dd5abe4c28f1 2023-04-01 03:57:24,469 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/e425ec5b623045308fdbf25bbbba3c50 2023-04-01 03:57:24,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:24,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:24,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321504698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321504736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321504714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321504737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321504740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321504839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321504842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321504843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321504845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,912 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/b3b8acb0bc5347348baf42beec29545e 2023-04-01 03:57:24,921 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2a20bd6e3c994cedbc92dd5abe4c28f1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1 2023-04-01 03:57:24,926 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 03:57:24,927 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/e425ec5b623045308fdbf25bbbba3c50 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50 2023-04-01 03:57:24,940 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 03:57:24,941 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/b3b8acb0bc5347348baf42beec29545e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e 2023-04-01 03:57:24,945 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 03:57:24,951 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 152d4d46bcbd27b89955640499bc0d72 in 937ms, sequenceid=67, compaction requested=false 2023-04-01 03:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:24,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:24,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:24,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:24,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611c6880[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:24,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:24,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:24,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:24,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:24,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:24,982 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,982 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:24,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:24,987 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:24,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:24,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:24,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:24,987 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:24,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:24,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,030 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,030 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,038 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,038 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:25,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:57:25,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:25,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321505058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321505061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321505062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321505064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,131 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,137 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,137 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,162 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,163 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,163 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b72c4c0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,165 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,165 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321505164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,167 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,168 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,168 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,168 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321505167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321505167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321505168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,197 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,198 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,298 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,309 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,312 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,316 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,316 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,316 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16407fe5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,329 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,329 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,332 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,335 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,336 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,336 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321505367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321505369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321505370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321505371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,436 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,440 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,445 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,445 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,445 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,446 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,447 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f317d9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,455 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,456 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,456 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,456 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,459 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,459 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,465 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,465 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/7a63e175a05546afaf2f70a7217a2777 2023-04-01 03:57:25,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/2e61240f8ec449d5ba0a017dee56c7a5 2023-04-01 03:57:25,565 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,568 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,568 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,568 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,568 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,568 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,568 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,569 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,573 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a29262f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,576 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,577 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,577 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,577 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,586 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,588 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,589 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,589 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,591 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,593 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,595 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,595 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,595 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321505672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321505672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321505673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321505674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,695 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,701 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,701 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,702 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,702 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,702 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,703 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,705 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,705 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,705 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,721 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,724 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,725 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0a76e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,727 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,736 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,736 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,740 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,740 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,740 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,771 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,771 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,871 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:25,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:25,878 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:25,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:25,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:25,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:25,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:25,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:25,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:25,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:25,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:25,886 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,886 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:25,886 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:25,886 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:25,888 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125a4f22[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:25,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:25,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:25,891 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,891 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:25,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:25,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:25,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:25,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:25,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:25,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:25,896 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:25,896 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:25,896 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:25,896 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:25,896 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:25,898 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,898 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,898 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:25,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:25,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:25,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:25,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:25,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:25,904 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:25,904 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:25,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:25,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:25,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=88 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/6c6937238c2d495c8a07bf9344f3baf3 2023-04-01 03:57:25,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/7a63e175a05546afaf2f70a7217a2777 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777 2023-04-01 03:57:25,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777, entries=150, sequenceid=88, filesize=11.6 K 2023-04-01 03:57:25,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/2e61240f8ec449d5ba0a017dee56c7a5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5 2023-04-01 03:57:25,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5, entries=150, sequenceid=88, filesize=11.6 K 2023-04-01 03:57:25,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/6c6937238c2d495c8a07bf9344f3baf3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3 2023-04-01 03:57:25,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3, entries=150, sequenceid=88, filesize=11.6 K 2023-04-01 03:57:25,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 152d4d46bcbd27b89955640499bc0d72 in 939ms, sequenceid=88, compaction requested=true 2023-04-01 03:57:25,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:25,985 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:25,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:25,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:25,985 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:25,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:25,986 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:25,986 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/A is initiating minor compaction (all files) 2023-04-01 03:57:25,986 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:25,986 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/B is initiating minor compaction (all files) 2023-04-01 03:57:25,986 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/B in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,986 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/58b7139775e448918deaa87a50f57cc3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.9 K 2023-04-01 03:57:25,990 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 58b7139775e448918deaa87a50f57cc3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:25,992 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e425ec5b623045308fdbf25bbbba3c50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680321442556 2023-04-01 03:57:25,992 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2e61240f8ec449d5ba0a017dee56c7a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:25,986 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/A in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:25,998 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b8ad8a48d8e945a296157671f18a4406, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.9 K 2023-04-01 03:57:25,999 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#B#compaction#216 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:25,999 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b8ad8a48d8e945a296157671f18a4406, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:25,999 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2a20bd6e3c994cedbc92dd5abe4c28f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680321442556 2023-04-01 03:57:25,999 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7a63e175a05546afaf2f70a7217a2777, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:26,005 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:26,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:26,006 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:26,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:26,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:26,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:26,022 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:26,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:26,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:26,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:26,023 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:26,023 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:26,028 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:26,028 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:26,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:26,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:26,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:26,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:26,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:26,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:26,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:26,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:26,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:26,044 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:26,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:26,045 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:26,102 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/a7ad8829e7064ca0a2173042e34fcc76 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/a7ad8829e7064ca0a2173042e34fcc76 2023-04-01 03:57:26,110 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/86d04a6829ef4b65b15349b2ed44b241 2023-04-01 03:57:26,112 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/B of 152d4d46bcbd27b89955640499bc0d72 into a7ad8829e7064ca0a2173042e34fcc76(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:26,112 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:26,112 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/B, priority=13, startTime=1680321445985; duration=0sec 2023-04-01 03:57:26,112 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:26,112 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:57:26,120 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:57:26,121 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/C is initiating minor compaction (all files) 2023-04-01 03:57:26,121 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/C in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:26,121 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9390bc8038284059ae9eb4fe1753d8c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=34.9 K 2023-04-01 03:57:26,127 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9390bc8038284059ae9eb4fe1753d8c0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680321442526 2023-04-01 03:57:26,127 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b3b8acb0bc5347348baf42beec29545e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680321442556 2023-04-01 03:57:26,127 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6c6937238c2d495c8a07bf9344f3baf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:26,141 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#C#compaction#219 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:26,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:26,184 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/0d3206bcc3f74867895adf9c6df6e4fe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/0d3206bcc3f74867895adf9c6df6e4fe 2023-04-01 03:57:26,189 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/C of 152d4d46bcbd27b89955640499bc0d72 into 0d3206bcc3f74867895adf9c6df6e4fe(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:26,189 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:26,189 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/C, priority=13, startTime=1680321445985; duration=0sec 2023-04-01 03:57:26,189 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:26,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321506201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321506205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321506206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321506205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321506307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321506307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321506308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321506308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,468 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/623e8f31b6ab488bb58f58a73055d4f2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/623e8f31b6ab488bb58f58a73055d4f2 2023-04-01 03:57:26,472 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/A of 152d4d46bcbd27b89955640499bc0d72 into 623e8f31b6ab488bb58f58a73055d4f2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:26,473 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:26,473 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/A, priority=13, startTime=1680321445985; duration=0sec 2023-04-01 03:57:26,473 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:26,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321506512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321506514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321506522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321506522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,579 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/708c414fd66a4392bbc8b948409238d8 2023-04-01 03:57:26,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321506754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,756 DEBUG [Thread-1610] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321506814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321506818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321506824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:26,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321506825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,011 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/463d360ae6464c5ca9ab44ee22cbb327 2023-04-01 03:57:27,018 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/86d04a6829ef4b65b15349b2ed44b241 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241 2023-04-01 03:57:27,027 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241, entries=150, sequenceid=102, filesize=11.6 K 2023-04-01 03:57:27,028 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/708c414fd66a4392bbc8b948409238d8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8 2023-04-01 03:57:27,043 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8, entries=150, sequenceid=102, filesize=11.6 K 2023-04-01 03:57:27,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/463d360ae6464c5ca9ab44ee22cbb327 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327 2023-04-01 03:57:27,049 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327, entries=150, sequenceid=102, filesize=11.6 K 2023-04-01 03:57:27,055 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 152d4d46bcbd27b89955640499bc0d72 in 1011ms, sequenceid=102, compaction requested=false 2023-04-01 03:57:27,055 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:27,055 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:27,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:27,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:27,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:27,055 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:27,057 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:27,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@287df6f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:27,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:27,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:27,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:27,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:27,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:27,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:27,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:27,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:27,063 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:27,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:27,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:27,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:27,063 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:27,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:27,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,112 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:27,112 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:27,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:27,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:27,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:27,113 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:27,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:27,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:27,114 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:27,114 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:27,214 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:27,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:27,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:27,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:27,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:27,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:27,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:27,220 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,220 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:27,222 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:27,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:27,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:27,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:27,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:27,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:27,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:27,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:27,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:27,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:27,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:27,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:27,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:27,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:27,234 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:57:27,234 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:27,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:27,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:27,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:27,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:27,235 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:27,270 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/06fc9e2bfe134acfb274f663dd215b24 2023-04-01 03:57:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:27,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:27,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321507339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321507340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321507341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321507340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321507443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321507445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321507446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321507455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321507648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321507648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321507654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321507665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,727 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/b5890957ea1843d89eb5726d989463a6 2023-04-01 03:57:27,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321507952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321507952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321507958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:27,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:27,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321507967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,171 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/619fe5103da94ea9b475fb1dd46e89e9 2023-04-01 03:57:28,177 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/06fc9e2bfe134acfb274f663dd215b24 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24 2023-04-01 03:57:28,182 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24, entries=150, sequenceid=123, filesize=11.6 K 2023-04-01 03:57:28,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/b5890957ea1843d89eb5726d989463a6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6 2023-04-01 03:57:28,188 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6, entries=150, sequenceid=123, filesize=11.6 K 2023-04-01 03:57:28,189 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/619fe5103da94ea9b475fb1dd46e89e9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9 2023-04-01 03:57:28,192 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9, entries=150, sequenceid=123, filesize=11.6 K 2023-04-01 03:57:28,193 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 959ms, sequenceid=123, compaction requested=true 2023-04-01 03:57:28,193 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:28,193 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:28,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:28,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:28,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:28,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:28,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:28,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:28,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:28,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@415c894e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:28,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:28,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:28,199 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:28,199 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:28,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,199 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:28,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:28,202 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:28,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:28,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:28,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:28,202 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:28,204 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,204 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,204 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:28,205 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:28,207 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:28,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:28,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:28,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:28,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:28,210 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:28,210 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:28,210 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,310 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:28,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:28,315 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:28,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:28,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:28,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:28,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:28,317 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:28,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:28,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:28,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:28,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:28,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:28,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:28,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:28,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:28,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:28,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:28,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:28,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:28,319 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:28,458 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:28,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321508475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321508476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321508476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321508476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321508578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321508579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321508579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321508579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,725 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/25c0d5434dc34e38ba6216513a738b5c 2023-04-01 03:57:28,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321508781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321508781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321508782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:28,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:28,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321508782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321509083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321509084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321509086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321509086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,134 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/2e4649b72cdc442782a1c37f3eaea447 2023-04-01 03:57:29,143 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/40e04bd6f6184a6c939710f2a6318eef 2023-04-01 03:57:29,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/25c0d5434dc34e38ba6216513a738b5c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c 2023-04-01 03:57:29,150 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c, entries=150, sequenceid=138, filesize=11.8 K 2023-04-01 03:57:29,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/2e4649b72cdc442782a1c37f3eaea447 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447 2023-04-01 03:57:29,154 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447, entries=150, sequenceid=138, filesize=11.8 K 2023-04-01 03:57:29,155 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/40e04bd6f6184a6c939710f2a6318eef as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef 2023-04-01 03:57:29,158 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef, entries=150, sequenceid=138, filesize=11.8 K 2023-04-01 03:57:29,158 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 152d4d46bcbd27b89955640499bc0d72 in 839ms, sequenceid=138, compaction requested=true 2023-04-01 03:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:29,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,160 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:29,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:29,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:29,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43159ddc[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:29,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:29,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:29,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:29,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:29,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:29,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:29,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:29,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:29,167 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:29,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:29,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:29,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:29,167 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:29,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:29,170 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,171 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:29,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,172 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:29,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:29,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:29,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:29,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:29,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:29,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:29,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:29,173 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:29,173 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:29,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:29,273 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:29,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:29,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:29,276 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:29,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:29,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:29,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:29,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:29,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,277 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:29,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:29,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,278 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:29,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:29,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:29,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:29,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:29,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:29,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:29,279 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:29,279 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:29,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:29,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:29,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321509597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321509598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321509600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321509600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,686 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/d2ef73fa353641c2968bc1c110efec22 2023-04-01 03:57:29,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321509701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321509701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321509703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321509704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321509902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321509903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321509906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:29,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:29,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321509906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,095 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/c4f97de2ac86405f986bd1569906e834 2023-04-01 03:57:30,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321510205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321510206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321510208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321510208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,503 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/1b3149d80a0b4d74bb291c12c65b9316 2023-04-01 03:57:30,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/d2ef73fa353641c2968bc1c110efec22 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22 2023-04-01 03:57:30,510 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22, entries=150, sequenceid=156, filesize=11.8 K 2023-04-01 03:57:30,511 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/c4f97de2ac86405f986bd1569906e834 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834 2023-04-01 03:57:30,514 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834, entries=150, sequenceid=156, filesize=11.8 K 2023-04-01 03:57:30,515 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/1b3149d80a0b4d74bb291c12c65b9316 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316 2023-04-01 03:57:30,525 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316, entries=150, sequenceid=156, filesize=11.8 K 2023-04-01 03:57:30,526 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 1247ms, sequenceid=156, compaction requested=true 2023-04-01 03:57:30,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:30,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:30,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:30,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:30,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:30,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:30,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:30,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:30,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:30,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f6f9c3[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:30,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:30,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:30,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,536 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:30,537 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:30,537 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:30,537 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:30,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:30,539 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:30,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:30,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:30,539 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:30,539 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:30,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:30,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:30,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:30,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:30,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:30,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:30,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:30,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:30,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:30,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:30,546 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:30,546 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:30,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:30,646 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:30,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:30,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:30,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:30,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:30,649 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:30,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:30,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:30,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:30,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:30,650 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:30,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:30,650 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:30,650 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,650 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:30,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:30,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:30,651 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:30,651 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:30,651 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:30,652 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:30,652 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:30,652 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:30,652 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:30,684 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/a0e8776156c64e4ab2899bcca859bf43 2023-04-01 03:57:30,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:30,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:30,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321510728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321510731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321510731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321510732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321510774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,778 DEBUG [Thread-1610] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8219 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:30,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321510833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321510836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321510837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:30,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321510838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321511036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321511040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321511040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321511040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,133 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/f24aa079d76b422d87d2264f26353c63 2023-04-01 03:57:31,142 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/01121558223e49d9b1b9288dbed062ce 2023-04-01 03:57:31,145 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/a0e8776156c64e4ab2899bcca859bf43 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43 2023-04-01 03:57:31,148 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43, entries=100, sequenceid=171, filesize=9.4 K 2023-04-01 03:57:31,149 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/f24aa079d76b422d87d2264f26353c63 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63 2023-04-01 03:57:31,154 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63, entries=100, sequenceid=171, filesize=9.4 K 2023-04-01 03:57:31,154 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/01121558223e49d9b1b9288dbed062ce as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce 2023-04-01 03:57:31,157 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce, entries=100, sequenceid=171, filesize=9.4 K 2023-04-01 03:57:31,158 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 152d4d46bcbd27b89955640499bc0d72 in 506ms, sequenceid=171, compaction requested=true 2023-04-01 03:57:31,158 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:31,158 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:31,160 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:31,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:31,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f362275[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:31,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:31,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:31,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:31,164 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:31,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:31,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:31,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:31,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,165 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:31,167 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:31,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:31,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:31,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:31,167 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:31,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:31,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,169 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:31,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:31,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:31,170 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:31,171 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:31,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:31,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:31,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:31,173 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:31,173 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:31,173 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:31,173 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:31,174 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:31,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,273 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:31,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:31,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:31,276 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:31,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:31,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:31,276 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:31,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,277 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:31,278 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,278 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:31,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:31,279 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,279 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:31,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:31,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:31,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:31,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:31,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:31,280 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:57:31,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:31,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:31,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:31,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:31,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:31,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:31,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:31,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:31,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321511354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321511357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321511358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321511358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321511459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321511462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321511462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321511462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321511662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321511666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321511666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321511667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,686 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/b254261097b04596ac801fe222d97a6d 2023-04-01 03:57:31,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321511966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321511969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321511969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:31,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:31,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321511970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,112 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/28d7edafdddb4938bfaaa965373fb8fb 2023-04-01 03:57:32,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:32,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321512472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:32,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321512473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:32,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321512474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:32,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321512475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,520 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/52f08020bf9e48919d396e9631882235 2023-04-01 03:57:32,524 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/b254261097b04596ac801fe222d97a6d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d 2023-04-01 03:57:32,527 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d, entries=150, sequenceid=188, filesize=11.8 K 2023-04-01 03:57:32,527 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/28d7edafdddb4938bfaaa965373fb8fb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb 2023-04-01 03:57:32,530 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb, entries=150, sequenceid=188, filesize=11.8 K 2023-04-01 03:57:32,531 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/52f08020bf9e48919d396e9631882235 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235 2023-04-01 03:57:32,534 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235, entries=150, sequenceid=188, filesize=11.8 K 2023-04-01 03:57:32,534 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 1254ms, sequenceid=188, compaction requested=true 2023-04-01 03:57:32,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:32,535 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:32,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:32,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:32,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:32,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:32,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:32,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a85da24[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:32,537 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:32,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:32,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:32,541 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:32,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:32,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:32,543 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:32,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:32,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:32,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:32,543 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:32,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:32,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:32,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:32,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:32,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:32,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:32,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:32,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:32,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:32,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:32,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,551 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:32,551 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:32,651 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:32,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:32,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:32,655 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:32,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:32,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:32,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:32,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:32,732 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:32,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:32,733 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,733 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:32,733 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:32,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:32,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:32,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:32,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:32,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:32,734 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:32,734 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:32,734 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:32,735 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:33,141 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/c40fd8cfe2a34ca9a4707f88a9e360e2 2023-04-01 03:57:33,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:33,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:33,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321513509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321513510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321513513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321513513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,549 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/74ad6584e611406596b55719e90cd696 2023-04-01 03:57:33,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321513614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321513614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321513618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321513618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57638 deadline: 1680321513817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321513817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57662 deadline: 1680321513821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:33,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57676 deadline: 1680321513821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,958 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/23bcb3cff55b4cceb9fd68168f4e18f1 2023-04-01 03:57:33,964 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/c40fd8cfe2a34ca9a4707f88a9e360e2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2 2023-04-01 03:57:33,967 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2, entries=150, sequenceid=203, filesize=11.8 K 2023-04-01 03:57:33,967 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/74ad6584e611406596b55719e90cd696 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696 2023-04-01 03:57:33,970 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696, entries=150, sequenceid=203, filesize=11.8 K 2023-04-01 03:57:33,971 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/23bcb3cff55b4cceb9fd68168f4e18f1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1 2023-04-01 03:57:33,974 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1, entries=150, sequenceid=203, filesize=11.8 K 2023-04-01 03:57:33,975 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 152d4d46bcbd27b89955640499bc0d72 in 1241ms, sequenceid=203, compaction requested=true 2023-04-01 03:57:33,975 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:33,975 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:33,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:33,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:33,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:33,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:33,980 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:33,980 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:33,980 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,980 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75115fd5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:33,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:33,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,985 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:33,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:33,986 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:33,988 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:33,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:33,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:33,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:33,988 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:33,989 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,989 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:33,989 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:33,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:33,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:33,989 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:33,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:33,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:33,992 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:33,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:33,992 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:33,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:33,993 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:34,093 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:34,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:34,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:34,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:34,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:34,095 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:34,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:34,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:34,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:34,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,096 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:34,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,097 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:34,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:34,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:34,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:34,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:34,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:34,098 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:34,098 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:34,099 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:34,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:34,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:34,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321514136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321514137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57616 deadline: 1680321514138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321514140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321514241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321514242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321514242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321514244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321514445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321514445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321514446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321514448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,505 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/82f43c7a736e42ad87266bc984a540e7 2023-04-01 03:57:34,532 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/8257d7fccc5146389c75a48df4588ffa 2023-04-01 03:57:34,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321514747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321514748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321514748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:34,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321514751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,947 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/7203f12dec394687894dec9d5ed0ad40 2023-04-01 03:57:34,951 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/82f43c7a736e42ad87266bc984a540e7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7 2023-04-01 03:57:34,954 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7, entries=150, sequenceid=220, filesize=11.8 K 2023-04-01 03:57:34,955 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/8257d7fccc5146389c75a48df4588ffa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa 2023-04-01 03:57:34,957 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa, entries=150, sequenceid=220, filesize=11.8 K 2023-04-01 03:57:34,958 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/7203f12dec394687894dec9d5ed0ad40 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40 2023-04-01 03:57:34,961 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40, entries=150, sequenceid=220, filesize=11.8 K 2023-04-01 03:57:34,962 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 864ms, sequenceid=220, compaction requested=true 2023-04-01 03:57:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:34,964 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:34,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:34,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419d8855[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:34,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:34,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,966 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:34,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:34,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:34,966 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:34,967 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:34,967 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:34,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,967 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:34,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:34,969 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:34,969 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:34,969 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:34,969 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:34,969 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:34,985 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,985 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,985 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:34,985 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:34,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:34,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:34,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:34,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:34,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:34,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:34,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:34,996 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:34,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:34,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:34,996 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:34,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:34,996 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:34,996 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,996 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:34,996 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,096 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:35,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:35,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:35,099 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:35,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:35,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:35,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:35,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:35,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:35,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:35,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:35,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:35,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:35,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:35,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:35,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:35,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:35,102 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:35,102 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:35,109 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/5f0b5585b8bb496ea164ae8fc2b361f5 2023-04-01 03:57:35,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:35,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:35,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321515272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321515272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321515272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321515273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321515377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321515377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321515377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321515377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,517 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/99ca0ded7ac74949a2fb382d87290394 2023-04-01 03:57:35,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321515581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321515581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321515582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321515583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321515884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321515885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321515885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:35,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321515886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,926 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/6ff1f365fd28480b8a9f025407d3a7a0 2023-04-01 03:57:35,929 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/5f0b5585b8bb496ea164ae8fc2b361f5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5 2023-04-01 03:57:35,933 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5, entries=150, sequenceid=235, filesize=11.8 K 2023-04-01 03:57:35,933 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/99ca0ded7ac74949a2fb382d87290394 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394 2023-04-01 03:57:35,936 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394, entries=150, sequenceid=235, filesize=11.8 K 2023-04-01 03:57:35,937 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/6ff1f365fd28480b8a9f025407d3a7a0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0 2023-04-01 03:57:35,940 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0, entries=150, sequenceid=235, filesize=11.8 K 2023-04-01 03:57:35,941 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 152d4d46bcbd27b89955640499bc0d72 in 839ms, sequenceid=235, compaction requested=true 2023-04-01 03:57:35,941 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:35,941 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:35,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:35,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:35,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:35,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,943 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:35,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:35,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb63f99[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:35,944 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:35,944 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,946 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:35,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,946 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:35,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:35,950 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:35,950 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:35,950 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:35,950 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:35,950 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:35,951 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,951 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:35,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,951 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:35,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,953 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:35,955 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:35,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:35,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:35,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:35,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:35,955 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:35,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:35,955 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:35,955 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:36,055 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:36,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:36,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:36,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:36,058 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:36,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:36,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:36,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:36,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:36,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:36,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:36,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:36,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:36,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:36,061 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:36,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:36,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:36,062 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:36,062 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:36,063 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:36,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:36,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:36,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321516399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321516401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321516402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321516402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,469 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/1f177f9069e04201b959cdfdcae049b0 2023-04-01 03:57:36,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321516503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321516507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321516509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321516510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321516707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321516710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321516713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:36,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321516713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:36,878 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/bf39ee361bdd4864b6d1ae34d9290201 2023-04-01 03:57:37,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321517010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321517012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321517017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321517018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,285 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/716e3d9fdd924f97a9bfbc6783c849b0 2023-04-01 03:57:37,289 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/1f177f9069e04201b959cdfdcae049b0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0 2023-04-01 03:57:37,292 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0, entries=150, sequenceid=253, filesize=11.8 K 2023-04-01 03:57:37,293 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/bf39ee361bdd4864b6d1ae34d9290201 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201 2023-04-01 03:57:37,296 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201, entries=150, sequenceid=253, filesize=11.8 K 2023-04-01 03:57:37,296 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/716e3d9fdd924f97a9bfbc6783c849b0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0 2023-04-01 03:57:37,300 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0, entries=150, sequenceid=253, filesize=11.8 K 2023-04-01 03:57:37,300 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 152d4d46bcbd27b89955640499bc0d72 in 1238ms, sequenceid=253, compaction requested=true 2023-04-01 03:57:37,300 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:37,300 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:37,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:37,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:37,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:37,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:37,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:37,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd688b1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:37,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:37,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,306 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:37,306 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,306 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:37,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:37,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:37,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:37,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:37,309 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:37,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:37,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:37,310 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:37,310 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:37,310 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,310 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:37,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,310 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:37,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,312 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:37,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:37,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:37,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:37,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:37,313 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:37,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:37,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,314 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:37,314 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:37,314 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,414 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:37,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:37,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:37,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:37,417 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:37,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:37,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:37,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:37,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:37,419 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:37,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:37,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:37,419 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:37,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:37,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:37,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:37,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:37,420 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:37,420 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:37,420 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:37,420 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:37,420 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:37,420 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:37,420 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:37,420 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:37,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:37,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:37,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321517538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321517539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321517539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321517539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321517644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321517644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321517644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321517644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,826 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/efad616b4add45b1816cff95298d06d2 2023-04-01 03:57:37,842 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/ff2a72fdd024424ea4b74bc89ec80e8f 2023-04-01 03:57:37,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321517849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321517849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321517849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:37,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321517850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321518154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321518155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321518156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321518156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,251 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/1d9e812e18ed4808b91c92d0ac43f5ad 2023-04-01 03:57:38,254 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/efad616b4add45b1816cff95298d06d2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2 2023-04-01 03:57:38,276 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2, entries=150, sequenceid=267, filesize=11.9 K 2023-04-01 03:57:38,277 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/ff2a72fdd024424ea4b74bc89ec80e8f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f 2023-04-01 03:57:38,280 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f, entries=150, sequenceid=267, filesize=11.9 K 2023-04-01 03:57:38,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/1d9e812e18ed4808b91c92d0ac43f5ad as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad 2023-04-01 03:57:38,284 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad, entries=150, sequenceid=267, filesize=11.9 K 2023-04-01 03:57:38,285 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 152d4d46bcbd27b89955640499bc0d72 in 865ms, sequenceid=267, compaction requested=true 2023-04-01 03:57:38,285 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:38,285 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:38,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:38,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:38,285 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,287 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:38,287 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:38,289 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:38,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:38,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c68fe9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:38,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,291 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:38,292 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,292 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:38,292 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:38,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:38,295 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:38,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:38,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:38,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:38,295 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:38,317 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:38,318 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:38,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:38,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:38,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:38,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:38,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:38,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:38,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:38,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,322 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:38,322 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:38,422 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:38,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:38,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:38,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:38,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:38,425 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:38,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:38,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:38,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:38,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:38,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:38,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:38,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:38,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:38,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:38,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:38,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:38,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:38,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:38,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:38,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:38,429 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:38,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:38,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:38,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321518671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321518676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321518676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321518677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321518778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321518784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321518784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:38,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321518784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,835 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/bb9f25c9e9224dc5ad3fabc0590d7ccb 2023-04-01 03:57:38,860 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/0f66c62fed2b4e2b8086afe28937de18 2023-04-01 03:57:38,873 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d 2023-04-01 03:57:38,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/bb9f25c9e9224dc5ad3fabc0590d7ccb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb 2023-04-01 03:57:38,882 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb, entries=150, sequenceid=285, filesize=11.9 K 2023-04-01 03:57:38,883 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/0f66c62fed2b4e2b8086afe28937de18 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18 2023-04-01 03:57:38,889 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18, entries=150, sequenceid=285, filesize=11.9 K 2023-04-01 03:57:38,890 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d 2023-04-01 03:57:38,894 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d, entries=150, sequenceid=285, filesize=11.9 K 2023-04-01 03:57:38,895 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 152d4d46bcbd27b89955640499bc0d72 in 467ms, sequenceid=285, compaction requested=true 2023-04-01 03:57:38,895 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:38,895 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:38,897 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,897 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,898 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,898 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29347004[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:38,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:38,899 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,900 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:38,900 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:38,900 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:38,900 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:38,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:38,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:38,902 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:38,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:38,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:38,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:38,902 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:38,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:38,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:38,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:38,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:38,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:38,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:38,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:38,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:38,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:38,906 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:38,906 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:38,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:38,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:38,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:38,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:38,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/6fbd6f03af884ad0adc3fb89a4dd4ee4 2023-04-01 03:57:39,006 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:39,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/c994a2a337d64e768b60908072369b8b 2023-04-01 03:57:39,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:39,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:39,012 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:39,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:39,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:39,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:39,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:39,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:39,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:39,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:39,015 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:39,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:39,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:39,016 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:39,017 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,017 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:39,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5700c84c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:39,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:39,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,020 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,021 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:39,021 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:39,021 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:39,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:39,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,045 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:39,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:39,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:39,049 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:39,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:39,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:39,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:39,049 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:39,062 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,062 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,062 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,063 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:39,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,072 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:39,073 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,173 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:39,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:39,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:39,176 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:39,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:39,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:39,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,177 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:39,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:39,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:39,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:39,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:39,184 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,184 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,184 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,184 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:39,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18dc511d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:39,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:39,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,186 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:39,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:39,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:39,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:39,190 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:39,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:39,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:39,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:39,190 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:39,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,197 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:39,197 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:39,297 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:39,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:39,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:39,300 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:39,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:39,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:39,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,302 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:39,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:39,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:39,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:39,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:39,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,304 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:39,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:39,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:39,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:39,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:39,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:39,309 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:39,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28552c28[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:39,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:39,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:39,314 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:39,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:39,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:39,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:39,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:39,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:39,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,323 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:39,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,326 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:39,326 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:39,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/74cc7d1de550401ebbe510eaf0a0d925 2023-04-01 03:57:39,426 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:39,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/6fbd6f03af884ad0adc3fb89a4dd4ee4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4 2023-04-01 03:57:39,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:39,431 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:39,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:39,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:39,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4, entries=200, sequenceid=300, filesize=14.3 K 2023-04-01 03:57:39,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/c994a2a337d64e768b60908072369b8b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b 2023-04-01 03:57:39,434 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:39,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:39,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:39,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:39,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:39,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:39,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b, entries=150, sequenceid=300, filesize=11.9 K 2023-04-01 03:57:39,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/74cc7d1de550401ebbe510eaf0a0d925 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925 2023-04-01 03:57:39,438 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,438 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,438 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226bcd76[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:39,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:39,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,440 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:39,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:39,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925, entries=150, sequenceid=300, filesize=11.9 K 2023-04-01 03:57:39,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 152d4d46bcbd27b89955640499bc0d72 in 455ms, sequenceid=300, compaction requested=true 2023-04-01 03:57:39,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:39,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:39,442 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 14 store files, 0 compacting, 14 eligible, 16 blocking 2023-04-01 03:57:39,442 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 14 store files, 0 compacting, 14 eligible, 16 blocking 2023-04-01 03:57:39,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:39,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:39,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:39,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:39,444 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:39,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:39,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:39,445 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:39,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117666 starting at candidate #1 after considering 68 permutations with 68 in ratio 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117666 starting at candidate #1 after considering 68 permutations with 68 in ratio 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/B is initiating minor compaction 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/A is initiating minor compaction 2023-04-01 03:57:39,448 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/B in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,448 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/A in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,448 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=114.9 K 2023-04-01 03:57:39,448 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=114.9 K 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 708c414fd66a4392bbc8b948409238d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102 2023-04-01 03:57:39,448 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 86d04a6829ef4b65b15349b2ed44b241, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 06fc9e2bfe134acfb274f663dd215b24, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 25c0d5434dc34e38ba6216513a738b5c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d2ef73fa353641c2968bc1c110efec22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting a0e8776156c64e4ab2899bcca859bf43, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=171 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b254261097b04596ac801fe222d97a6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c40fd8cfe2a34ca9a4707f88a9e360e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 82f43c7a736e42ad87266bc984a540e7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5f0b5585b8bb496ea164ae8fc2b361f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1f177f9069e04201b959cdfdcae049b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253 2023-04-01 03:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b5890957ea1843d89eb5726d989463a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2e4649b72cdc442782a1c37f3eaea447, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c4f97de2ac86405f986bd1569906e834, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f24aa079d76b422d87d2264f26353c63, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=171 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 28d7edafdddb4938bfaaa965373fb8fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188 2023-04-01 03:57:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,449 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 74ad6584e611406596b55719e90cd696, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203 2023-04-01 03:57:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,450 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8257d7fccc5146389c75a48df4588ffa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220 2023-04-01 03:57:39,450 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 99ca0ded7ac74949a2fb382d87290394, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235 2023-04-01 03:57:39,450 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bf39ee361bdd4864b6d1ae34d9290201, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253 2023-04-01 03:57:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:39,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:39,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:39,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,455 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:39,455 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:39,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:39,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:39,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:39,465 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#A#compaction#258 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:39,468 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#B#compaction#259 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:39,490 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/22f1715d28a94a6e97b5dfd16216e70b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/22f1715d28a94a6e97b5dfd16216e70b 2023-04-01 03:57:39,492 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/b48b82e438cc49bd8ff829e6c0befb66 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b48b82e438cc49bd8ff829e6c0befb66 2023-04-01 03:57:39,495 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 152d4d46bcbd27b89955640499bc0d72/A of 152d4d46bcbd27b89955640499bc0d72 into 22f1715d28a94a6e97b5dfd16216e70b(size=12.1 K), total size for store is 62.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:39,495 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:39,495 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/A, priority=2, startTime=1680321459442; duration=0sec 2023-04-01 03:57:39,495 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:39,495 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 14 store files, 0 compacting, 14 eligible, 16 blocking 2023-04-01 03:57:39,498 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117666 starting at candidate #1 after considering 68 permutations with 68 in ratio 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/C is initiating minor compaction 2023-04-01 03:57:39,499 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/C in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,499 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=114.9 K 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 463d360ae6464c5ca9ab44ee22cbb327, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 619fe5103da94ea9b475fb1dd46e89e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 40e04bd6f6184a6c939710f2a6318eef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1b3149d80a0b4d74bb291c12c65b9316, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 01121558223e49d9b1b9288dbed062ce, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=171 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 52f08020bf9e48919d396e9631882235, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 23bcb3cff55b4cceb9fd68168f4e18f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7203f12dec394687894dec9d5ed0ad40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6ff1f365fd28480b8a9f025407d3a7a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235 2023-04-01 03:57:39,499 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 716e3d9fdd924f97a9bfbc6783c849b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253 2023-04-01 03:57:39,502 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 152d4d46bcbd27b89955640499bc0d72/B of 152d4d46bcbd27b89955640499bc0d72 into b48b82e438cc49bd8ff829e6c0befb66(size=12.1 K), total size for store is 59.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:39,502 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:39,502 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/B, priority=2, startTime=1680321459442; duration=0sec 2023-04-01 03:57:39,502 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:39,517 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#C#compaction#260 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:39,539 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/459391686a044c73b22804931ae1b3da as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/459391686a044c73b22804931ae1b3da 2023-04-01 03:57:39,544 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 152d4d46bcbd27b89955640499bc0d72/C of 152d4d46bcbd27b89955640499bc0d72 into 459391686a044c73b22804931ae1b3da(size=12.1 K), total size for store is 59.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:39,544 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:39,544 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/C, priority=2, startTime=1680321459442; duration=0sec 2023-04-01 03:57:39,544 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:39,555 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:39,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:39,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:39,559 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:39,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:39,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:39,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:39,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:39,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:39,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:39,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:39,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:39,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:39,562 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:39,562 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:39,562 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:39,566 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:39,566 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:39,566 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:39,566 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:39,592 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/b4c2c9bedc8442a0a4d610cc2cad9ec3 2023-04-01 03:57:39,608 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/8028edc0870844ac9de8a1fae4e1e531 2023-04-01 03:57:39,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:39,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:39,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321519972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321519977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321519977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:39,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:39,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321519987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,020 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/2727ad9dc7fa477781c60b7a256f73a5 2023-04-01 03:57:40,024 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/b4c2c9bedc8442a0a4d610cc2cad9ec3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3 2023-04-01 03:57:40,030 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3, entries=150, sequenceid=320, filesize=11.9 K 2023-04-01 03:57:40,031 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/8028edc0870844ac9de8a1fae4e1e531 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531 2023-04-01 03:57:40,035 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531, entries=150, sequenceid=320, filesize=11.9 K 2023-04-01 03:57:40,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/2727ad9dc7fa477781c60b7a256f73a5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5 2023-04-01 03:57:40,042 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5, entries=150, sequenceid=320, filesize=11.9 K 2023-04-01 03:57:40,043 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 152d4d46bcbd27b89955640499bc0d72 in 477ms, sequenceid=320, compaction requested=true 2023-04-01 03:57:40,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:40,043 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:40,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:40,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:40,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:40,045 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737833df[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:40,046 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:40,046 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,048 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:40,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:40,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:40,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:40,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:40,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:40,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:40,054 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:40,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:40,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:40,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:40,055 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:40,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:40,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,057 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:40,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:40,061 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:40,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:40,061 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:40,161 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:40,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:40,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:40,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:40,166 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:40,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:40,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:40,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,168 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,169 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,169 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,169 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:40,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:40,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:40,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:40,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:40,174 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:40,174 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:40,174 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,174 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:40,174 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:57:40,174 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:40,174 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,175 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:40,175 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,175 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:40,175 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,209 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/abad97d623b14238b31eaa296777a605 2023-04-01 03:57:40,244 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/acfc3256801a480aaf15de55351de362 2023-04-01 03:57:40,264 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/457a7647297f46f7b55b107e119dccd7 2023-04-01 03:57:40,274 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/abad97d623b14238b31eaa296777a605 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605 2023-04-01 03:57:40,278 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 03:57:40,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/acfc3256801a480aaf15de55351de362 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362 2023-04-01 03:57:40,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:40,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:40,284 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 03:57:40,290 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/457a7647297f46f7b55b107e119dccd7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7 2023-04-01 03:57:40,296 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 03:57:40,298 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=53.67 KB/54960 for 152d4d46bcbd27b89955640499bc0d72 in 124ms, sequenceid=335, compaction requested=true 2023-04-01 03:57:40,298 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:40,298 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,298 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:40,298 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:40,298 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:40,298 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:40,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:40,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:40,302 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,302 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,303 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,303 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:40,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d6c014[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:40,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:40,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,306 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,306 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,306 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:40,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:40,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:40,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:40,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:40,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:40,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:40,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:40,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:40,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:40,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:40,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:40,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:40,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:40,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:40,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,320 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:40,324 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:40,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:40,324 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:40,324 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:40,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:40,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:40,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321520340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321520340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321520341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321520342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2f589b79b6384959baf4965030d00389 2023-04-01 03:57:40,425 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:40,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:40,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/f03501f8fdd3406f93aa88646eb5f627 2023-04-01 03:57:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:40,446 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:40,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:40,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:40,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:40,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321520449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321520449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321520449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321520450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:40,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:40,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:40,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:40,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:40,477 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,477 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:40,477 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:40,477 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:40,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:40,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,479 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,479 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:40,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@863996a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:40,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:40,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/c81229c498b547f5ade8db10c7bbf804 2023-04-01 03:57:40,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:40,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:40,483 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:40,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,484 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:40,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/2f589b79b6384959baf4965030d00389 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389 2023-04-01 03:57:40,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:40,487 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:40,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:40,491 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:40,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:40,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:40,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:40,491 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:40,493 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:40,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:40,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:40,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,499 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:40,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:40,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:40,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:40,500 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:40,500 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:40,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:40,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:40,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:40,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389, entries=250, sequenceid=347, filesize=16.7 K 2023-04-01 03:57:40,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/f03501f8fdd3406f93aa88646eb5f627 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627 2023-04-01 03:57:40,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627, entries=150, sequenceid=347, filesize=11.9 K 2023-04-01 03:57:40,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/c81229c498b547f5ade8db10c7bbf804 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804 2023-04-01 03:57:40,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804, entries=150, sequenceid=347, filesize=11.9 K 2023-04-01 03:57:40,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 152d4d46bcbd27b89955640499bc0d72 in 251ms, sequenceid=347, compaction requested=true 2023-04-01 03:57:40,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:40,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:40,549 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 03:57:40,549 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 03:57:40,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:40,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:40,553 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 104906 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 03:57:40,553 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/A is initiating minor compaction (all files) 2023-04-01 03:57:40,553 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/A in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,553 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/623e8f31b6ab488bb58f58a73055d4f2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/22f1715d28a94a6e97b5dfd16216e70b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=102.4 K 2023-04-01 03:57:40,553 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 623e8f31b6ab488bb58f58a73055d4f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:40,553 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97586 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 03:57:40,554 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/B is initiating minor compaction (all files) 2023-04-01 03:57:40,554 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/B in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 22f1715d28a94a6e97b5dfd16216e70b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680321455267 2023-04-01 03:57:40,554 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/a7ad8829e7064ca0a2173042e34fcc76, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b48b82e438cc49bd8ff829e6c0befb66, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=95.3 K 2023-04-01 03:57:40,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting efad616b4add45b1816cff95298d06d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680321456400 2023-04-01 03:57:40,554 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a7ad8829e7064ca0a2173042e34fcc76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:40,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting bb9f25c9e9224dc5ad3fabc0590d7ccb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680321457531 2023-04-01 03:57:40,555 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6fbd6f03af884ad0adc3fb89a4dd4ee4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680321458664 2023-04-01 03:57:40,555 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b48b82e438cc49bd8ff829e6c0befb66, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680321455267 2023-04-01 03:57:40,555 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b4c2c9bedc8442a0a4d610cc2cad9ec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321459010 2023-04-01 03:57:40,555 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting abad97d623b14238b31eaa296777a605, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680321459660 2023-04-01 03:57:40,556 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ff2a72fdd024424ea4b74bc89ec80e8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680321456400 2023-04-01 03:57:40,556 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2f589b79b6384959baf4965030d00389, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460292 2023-04-01 03:57:40,556 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0f66c62fed2b4e2b8086afe28937de18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680321457531 2023-04-01 03:57:40,556 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c994a2a337d64e768b60908072369b8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680321458675 2023-04-01 03:57:40,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8028edc0870844ac9de8a1fae4e1e531, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321459010 2023-04-01 03:57:40,557 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting acfc3256801a480aaf15de55351de362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680321459660 2023-04-01 03:57:40,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f03501f8fdd3406f93aa88646eb5f627, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460294 2023-04-01 03:57:40,581 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#A#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:40,592 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#B#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:40,600 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:40,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:40,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:40,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:40,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:40,614 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:40,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:40,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:40,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,615 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:40,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,617 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:40,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:40,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:40,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:40,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:40,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:40,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:40,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:40,634 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:40,634 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,634 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:40,634 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:40,634 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:57:40,634 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:40,634 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,634 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:40,635 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,635 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:40,635 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:40,639 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/ca0b9c8952e64028ba9402300ac158c3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ca0b9c8952e64028ba9402300ac158c3 2023-04-01 03:57:40,644 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/B of 152d4d46bcbd27b89955640499bc0d72 into ca0b9c8952e64028ba9402300ac158c3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:40,645 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:40,645 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/B, priority=8, startTime=1680321460549; duration=0sec 2023-04-01 03:57:40,645 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:40,645 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 03:57:40,647 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97586 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 03:57:40,648 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/C is initiating minor compaction (all files) 2023-04-01 03:57:40,648 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/C in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:40,648 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/0d3206bcc3f74867895adf9c6df6e4fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/459391686a044c73b22804931ae1b3da, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=95.3 K 2023-04-01 03:57:40,648 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0d3206bcc3f74867895adf9c6df6e4fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=88, earliestPutTs=1680321444697 2023-04-01 03:57:40,648 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 459391686a044c73b22804931ae1b3da, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680321455267 2023-04-01 03:57:40,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1d9e812e18ed4808b91c92d0ac43f5ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680321456400 2023-04-01 03:57:40,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ba3e7c3b2cbe4c15ac7583ff90d35d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680321457531 2023-04-01 03:57:40,649 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 74cc7d1de550401ebbe510eaf0a0d925, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680321458675 2023-04-01 03:57:40,651 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2727ad9dc7fa477781c60b7a256f73a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680321459010 2023-04-01 03:57:40,651 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 457a7647297f46f7b55b107e119dccd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680321459660 2023-04-01 03:57:40,652 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c81229c498b547f5ade8db10c7bbf804, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460294 2023-04-01 03:57:40,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:40,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:40,672 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/c8b1f4d23fc64bbc9b936927894b61cc 2023-04-01 03:57:40,680 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#C#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:40,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321520696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321520696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,702 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/7b74c5293d10493b90426919de2ac3c5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7b74c5293d10493b90426919de2ac3c5 2023-04-01 03:57:40,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321520696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321520701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,712 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/C of 152d4d46bcbd27b89955640499bc0d72 into 7b74c5293d10493b90426919de2ac3c5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:40,713 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:40,713 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/C, priority=8, startTime=1680321460549; duration=0sec 2023-04-01 03:57:40,713 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:40,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57598 deadline: 1680321520802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321520802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321520803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,806 DEBUG [Thread-1610] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18247 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:57:40,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321520803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:40,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321520814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321521006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321521007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321521007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,016 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/c82bd0c741854149a993c7083c80ab87 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c82bd0c741854149a993c7083c80ab87 2023-04-01 03:57:41,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321521017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,034 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/A of 152d4d46bcbd27b89955640499bc0d72 into c82bd0c741854149a993c7083c80ab87(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:41,034 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:41,034 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/A, priority=8, startTime=1680321460549; duration=0sec 2023-04-01 03:57:41,034 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:41,118 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/1ba0b61d26344c5d9e287d3d82efd38e 2023-04-01 03:57:41,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321521310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321521311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321521312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321521321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,536 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/bda7da5393a34dada1885fa21b266625 2023-04-01 03:57:41,539 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/c8b1f4d23fc64bbc9b936927894b61cc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc 2023-04-01 03:57:41,546 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:57:41,547 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/1ba0b61d26344c5d9e287d3d82efd38e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e 2023-04-01 03:57:41,550 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:57:41,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/bda7da5393a34dada1885fa21b266625 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625 2023-04-01 03:57:41,554 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625, entries=150, sequenceid=367, filesize=11.9 K 2023-04-01 03:57:41,555 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 152d4d46bcbd27b89955640499bc0d72 in 921ms, sequenceid=367, compaction requested=false 2023-04-01 03:57:41,555 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:41,555 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:41,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:41,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:41,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:41,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:41,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:41,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:41,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:41,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749ff56d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:41,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:41,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:41,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:41,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:41,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:41,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:41,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:41,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:41,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:41,561 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:41,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:41,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:41,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:41,561 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:41,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:41,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:41,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:41,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:41,574 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,576 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:41,577 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:41,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:41,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:41,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,578 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,578 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:41,578 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:41,678 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:41,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:41,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:41,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:41,681 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:41,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:41,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:41,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:41,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:41,683 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:41,683 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:41,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:41,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:41,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:41,684 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:41,684 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:41,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:41,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:41,685 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. started... 2023-04-01 03:57:41,685 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:41,685 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:41,703 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/9f70e11b5fde4592b3fe3c2ff619d911 2023-04-01 03:57:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:41,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. as already flushing 2023-04-01 03:57:41,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321521839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321521840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321521840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321521841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321521942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321521945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321521945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321521945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:41,978 DEBUG [Thread-1625] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f634c21 to 127.0.0.1:50457 2023-04-01 03:57:41,978 DEBUG [Thread-1625] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:41,979 DEBUG [Thread-1621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6cc75c6c to 127.0.0.1:50457 2023-04-01 03:57:41,979 DEBUG [Thread-1621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:41,981 DEBUG [Thread-1629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14784858 to 127.0.0.1:50457 2023-04-01 03:57:41,982 DEBUG [Thread-1629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:41,982 DEBUG [Thread-1623] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2bf0999a to 127.0.0.1:50457 2023-04-01 03:57:41,982 DEBUG [Thread-1623] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:41,984 DEBUG [Thread-1627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x488456ec to 127.0.0.1:50457 2023-04-01 03:57:41,984 DEBUG [Thread-1627] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:42,115 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/d45ef7d9fba544ebaf32f46d4d5843c0 2023-04-01 03:57:42,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321522146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321522149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321522149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321522150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57638 deadline: 1680321522448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57662 deadline: 1680321522451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:42,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57676 deadline: 1680321522451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57616 deadline: 1680321522451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,525 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/4829ca5a06434a8e94f196a9ae626574 2023-04-01 03:57:42,529 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/9f70e11b5fde4592b3fe3c2ff619d911 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911 2023-04-01 03:57:42,533 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911, entries=100, sequenceid=382, filesize=9.5 K 2023-04-01 03:57:42,533 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/d45ef7d9fba544ebaf32f46d4d5843c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0 2023-04-01 03:57:42,537 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0, entries=100, sequenceid=382, filesize=9.5 K 2023-04-01 03:57:42,538 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/4829ca5a06434a8e94f196a9ae626574 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574 2023-04-01 03:57:42,541 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574, entries=100, sequenceid=382, filesize=9.5 K 2023-04-01 03:57:42,542 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 152d4d46bcbd27b89955640499bc0d72 in 857ms, sequenceid=382, compaction requested=true 2023-04-01 03:57:42,542 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:42,542 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:42,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:42,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:42,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:42,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:42,544 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:42,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:42,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:42,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464bd6ea[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:42,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:42,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:42,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:42,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,547 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:42,547 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:42,548 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:42,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:42,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:42,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:42,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:42,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:42,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:42,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:42,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:42,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:42,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:42,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:42,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:42,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:42,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:42,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:42,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:42,556 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:42,556 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:42,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:42,557 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:42,557 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:42,557 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,557 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:42,557 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:42,557 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:42,657 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:42,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:42,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:42,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-01 03:57:42,953 DEBUG [Thread-1616] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x229d4b93 to 127.0.0.1:50457 2023-04-01 03:57:42,953 DEBUG [Thread-1616] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:42,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:42,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:42,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:42,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:42,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:42,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:57:42,953 DEBUG [Thread-1614] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3090fec3 to 127.0.0.1:50457 2023-04-01 03:57:42,954 DEBUG [Thread-1614] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:42,955 DEBUG [Thread-1612] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x553863db to 127.0.0.1:50457 2023-04-01 03:57:42,955 DEBUG [Thread-1612] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:42,956 DEBUG [Thread-1618] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5dc8411c to 127.0.0.1:50457 2023-04-01 03:57:42,956 DEBUG [Thread-1618] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:43,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/497d48c6748e43c6bb670022e03f37bc 2023-04-01 03:57:43,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/43e1ce33dc6e41a79ee39c420868d49a 2023-04-01 03:57:44,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/4a6a1828f27e4f0aba8d93035e2cfa52 2023-04-01 03:57:44,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/497d48c6748e43c6bb670022e03f37bc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc 2023-04-01 03:57:44,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc, entries=150, sequenceid=404, filesize=11.9 K 2023-04-01 03:57:44,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/43e1ce33dc6e41a79ee39c420868d49a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a 2023-04-01 03:57:44,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a, entries=150, sequenceid=404, filesize=11.9 K 2023-04-01 03:57:44,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/4a6a1828f27e4f0aba8d93035e2cfa52 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52 2023-04-01 03:57:44,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52, entries=150, sequenceid=404, filesize=11.9 K 2023-04-01 03:57:44,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=13.42 KB/13740 for 152d4d46bcbd27b89955640499bc0d72 in 1244ms, sequenceid=404, compaction requested=true 2023-04-01 03:57:44,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:44,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:44,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:44,197 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:44,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:44,197 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:44,198 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:44,198 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:44,198 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/A is initiating minor compaction (all files) 2023-04-01 03:57:44,198 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/B is initiating minor compaction (all files) 2023-04-01 03:57:44,198 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/A in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:44,198 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/B in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:44,198 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ca0b9c8952e64028ba9402300ac158c3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=46.0 K 2023-04-01 03:57:44,198 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c82bd0c741854149a993c7083c80ab87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=46.0 K 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ca0b9c8952e64028ba9402300ac158c3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460294 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c82bd0c741854149a993c7083c80ab87, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460294 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1ba0b61d26344c5d9e287d3d82efd38e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321460333 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c8b1f4d23fc64bbc9b936927894b61cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321460333 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d45ef7d9fba544ebaf32f46d4d5843c0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680321460659 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9f70e11b5fde4592b3fe3c2ff619d911, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680321460659 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 43e1ce33dc6e41a79ee39c420868d49a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680321461834 2023-04-01 03:57:44,199 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 497d48c6748e43c6bb670022e03f37bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680321461834 2023-04-01 03:57:44,206 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#A#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:44,206 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:44,620 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/54eeaf9a371e4265ab2e0b72a93ac40b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/54eeaf9a371e4265ab2e0b72a93ac40b 2023-04-01 03:57:44,620 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/17f1240abcf246aeab5acf24ba93f685 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/17f1240abcf246aeab5acf24ba93f685 2023-04-01 03:57:44,625 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/B of 152d4d46bcbd27b89955640499bc0d72 into 17f1240abcf246aeab5acf24ba93f685(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:44,625 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/A of 152d4d46bcbd27b89955640499bc0d72 into 54eeaf9a371e4265ab2e0b72a93ac40b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:44,625 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:44,625 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:44,625 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/B, priority=12, startTime=1680321464197; duration=0sec 2023-04-01 03:57:44,625 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/A, priority=12, startTime=1680321464197; duration=0sec 2023-04-01 03:57:44,625 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:57:44,625 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:44,625 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:57:44,627 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:57:44,627 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 152d4d46bcbd27b89955640499bc0d72/C is initiating minor compaction (all files) 2023-04-01 03:57:44,627 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 152d4d46bcbd27b89955640499bc0d72/C in TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:44,627 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7b74c5293d10493b90426919de2ac3c5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp, totalSize=46.0 K 2023-04-01 03:57:44,627 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7b74c5293d10493b90426919de2ac3c5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680321460294 2023-04-01 03:57:44,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bda7da5393a34dada1885fa21b266625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680321460333 2023-04-01 03:57:44,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4829ca5a06434a8e94f196a9ae626574, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680321460659 2023-04-01 03:57:44,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4a6a1828f27e4f0aba8d93035e2cfa52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680321461834 2023-04-01 03:57:44,637 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 152d4d46bcbd27b89955640499bc0d72#C#compaction#284 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:57:45,046 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/9a1d022f827e471e901af67c342dcf26 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9a1d022f827e471e901af67c342dcf26 2023-04-01 03:57:45,050 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 152d4d46bcbd27b89955640499bc0d72/C of 152d4d46bcbd27b89955640499bc0d72 into 9a1d022f827e471e901af67c342dcf26(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:57:45,050 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:45,050 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72., storeName=152d4d46bcbd27b89955640499bc0d72/C, priority=12, startTime=1680321464197; duration=0sec 2023-04-01 03:57:45,050 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:57:50,902 DEBUG [Thread-1610] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72d06f87 to 127.0.0.1:50457 2023-04-01 03:57:50,902 DEBUG [Thread-1610] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 4220 2023-04-01 03:57:50,902 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 4125 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 4332 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 4270 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 4186 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:57:50,903 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10f967f5 to 127.0.0.1:50457 2023-04-01 03:57:50,903 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:57:50,903 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:57:50,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:57:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:50,907 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321470907"}]},"ts":"1680321470907"} 2023-04-01 03:57:50,908 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:57:50,916 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:57:50,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, UNASSIGN}] 2023-04-01 03:57:50,918 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, UNASSIGN 2023-04-01 03:57:50,919 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=152d4d46bcbd27b89955640499bc0d72, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:50,919 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321470918"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321470918"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321470918"}]},"ts":"1680321470918"} 2023-04-01 03:57:50,919 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:51,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:51,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 152d4d46bcbd27b89955640499bc0d72, disabling compactions & flushes 2023-04-01 03:57:51,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:51,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:51,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. after waiting 0 ms 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:51,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 152d4d46bcbd27b89955640499bc0d72 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=A 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=B 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 152d4d46bcbd27b89955640499bc0d72, store=C 2023-04-01 03:57:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:51,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:51,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/986873c3348043788eccf4d94591d0bc 2023-04-01 03:57:51,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/b5d5c855d4b44dc5a05fd621565884e4 2023-04-01 03:57:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:51,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/cf73f66b15b94f2e8977c75408b633d3 2023-04-01 03:57:51,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/A/986873c3348043788eccf4d94591d0bc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/986873c3348043788eccf4d94591d0bc 2023-04-01 03:57:51,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/986873c3348043788eccf4d94591d0bc, entries=100, sequenceid=413, filesize=9.5 K 2023-04-01 03:57:51,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/B/b5d5c855d4b44dc5a05fd621565884e4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5d5c855d4b44dc5a05fd621565884e4 2023-04-01 03:57:51,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5d5c855d4b44dc5a05fd621565884e4, entries=100, sequenceid=413, filesize=9.5 K 2023-04-01 03:57:51,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/.tmp/C/cf73f66b15b94f2e8977c75408b633d3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/cf73f66b15b94f2e8977c75408b633d3 2023-04-01 03:57:51,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/cf73f66b15b94f2e8977c75408b633d3, entries=100, sequenceid=413, filesize=9.5 K 2023-04-01 03:57:51,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 152d4d46bcbd27b89955640499bc0d72 in 856ms, sequenceid=413, compaction requested=false 2023-04-01 03:57:51,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b8ad8a48d8e945a296157671f18a4406, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/623e8f31b6ab488bb58f58a73055d4f2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/22f1715d28a94a6e97b5dfd16216e70b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c82bd0c741854149a993c7083c80ab87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc] to archive 2023-04-01 03:57:51,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:51,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2e364b5662334f46a717f8e65662da31 2023-04-01 03:57:51,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5ab17cc8eb5644b1b018a5f9dcff5f67 2023-04-01 03:57:51,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b8ad8a48d8e945a296157671f18a4406 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b8ad8a48d8e945a296157671f18a4406 2023-04-01 03:57:51,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/15bba15550be409cb408a5c3a3f9a628 2023-04-01 03:57:51,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2a20bd6e3c994cedbc92dd5abe4c28f1 2023-04-01 03:57:51,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/623e8f31b6ab488bb58f58a73055d4f2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/623e8f31b6ab488bb58f58a73055d4f2 2023-04-01 03:57:51,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/7a63e175a05546afaf2f70a7217a2777 2023-04-01 03:57:51,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/86d04a6829ef4b65b15349b2ed44b241 2023-04-01 03:57:51,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/06fc9e2bfe134acfb274f663dd215b24 2023-04-01 03:57:51,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/25c0d5434dc34e38ba6216513a738b5c 2023-04-01 03:57:51,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/d2ef73fa353641c2968bc1c110efec22 2023-04-01 03:57:51,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/a0e8776156c64e4ab2899bcca859bf43 2023-04-01 03:57:51,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b254261097b04596ac801fe222d97a6d 2023-04-01 03:57:51,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c40fd8cfe2a34ca9a4707f88a9e360e2 2023-04-01 03:57:51,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/82f43c7a736e42ad87266bc984a540e7 2023-04-01 03:57:51,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/5f0b5585b8bb496ea164ae8fc2b361f5 2023-04-01 03:57:51,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/22f1715d28a94a6e97b5dfd16216e70b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/22f1715d28a94a6e97b5dfd16216e70b 2023-04-01 03:57:51,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/1f177f9069e04201b959cdfdcae049b0 2023-04-01 03:57:51,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/efad616b4add45b1816cff95298d06d2 2023-04-01 03:57:51,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/bb9f25c9e9224dc5ad3fabc0590d7ccb 2023-04-01 03:57:51,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/6fbd6f03af884ad0adc3fb89a4dd4ee4 2023-04-01 03:57:51,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/b4c2c9bedc8442a0a4d610cc2cad9ec3 2023-04-01 03:57:51,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/abad97d623b14238b31eaa296777a605 2023-04-01 03:57:51,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/2f589b79b6384959baf4965030d00389 2023-04-01 03:57:51,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c82bd0c741854149a993c7083c80ab87 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c82bd0c741854149a993c7083c80ab87 2023-04-01 03:57:51,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/c8b1f4d23fc64bbc9b936927894b61cc 2023-04-01 03:57:51,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/9f70e11b5fde4592b3fe3c2ff619d911 2023-04-01 03:57:51,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/497d48c6748e43c6bb670022e03f37bc 2023-04-01 03:57:51,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/58b7139775e448918deaa87a50f57cc3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/a7ad8829e7064ca0a2173042e34fcc76, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b48b82e438cc49bd8ff829e6c0befb66, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ca0b9c8952e64028ba9402300ac158c3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a] to archive 2023-04-01 03:57:51,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:51,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/26cf416f9c604b359d647ae0d82dfa43 2023-04-01 03:57:52,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0ea0182f57a740d49fd12dae94e2d28e 2023-04-01 03:57:52,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/58b7139775e448918deaa87a50f57cc3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/58b7139775e448918deaa87a50f57cc3 2023-04-01 03:57:52,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/5d7bb1e32b1a4c09ac17511f3be9a82e 2023-04-01 03:57:52,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/e425ec5b623045308fdbf25bbbba3c50 2023-04-01 03:57:52,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/a7ad8829e7064ca0a2173042e34fcc76 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/a7ad8829e7064ca0a2173042e34fcc76 2023-04-01 03:57:52,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e61240f8ec449d5ba0a017dee56c7a5 2023-04-01 03:57:52,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/708c414fd66a4392bbc8b948409238d8 2023-04-01 03:57:52,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5890957ea1843d89eb5726d989463a6 2023-04-01 03:57:52,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/2e4649b72cdc442782a1c37f3eaea447 2023-04-01 03:57:52,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c4f97de2ac86405f986bd1569906e834 2023-04-01 03:57:52,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f24aa079d76b422d87d2264f26353c63 2023-04-01 03:57:52,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:52,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/28d7edafdddb4938bfaaa965373fb8fb 2023-04-01 03:57:52,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/74ad6584e611406596b55719e90cd696 2023-04-01 03:57:52,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8257d7fccc5146389c75a48df4588ffa 2023-04-01 03:57:52,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/99ca0ded7ac74949a2fb382d87290394 2023-04-01 03:57:52,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b48b82e438cc49bd8ff829e6c0befb66 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b48b82e438cc49bd8ff829e6c0befb66 2023-04-01 03:57:52,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/bf39ee361bdd4864b6d1ae34d9290201 2023-04-01 03:57:52,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ff2a72fdd024424ea4b74bc89ec80e8f 2023-04-01 03:57:52,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/0f66c62fed2b4e2b8086afe28937de18 2023-04-01 03:57:52,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/c994a2a337d64e768b60908072369b8b 2023-04-01 03:57:52,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/8028edc0870844ac9de8a1fae4e1e531 2023-04-01 03:57:52,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/acfc3256801a480aaf15de55351de362 2023-04-01 03:57:52,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ca0b9c8952e64028ba9402300ac158c3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/ca0b9c8952e64028ba9402300ac158c3 2023-04-01 03:57:52,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/f03501f8fdd3406f93aa88646eb5f627 2023-04-01 03:57:52,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/1ba0b61d26344c5d9e287d3d82efd38e 2023-04-01 03:57:52,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/d45ef7d9fba544ebaf32f46d4d5843c0 2023-04-01 03:57:52,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/43e1ce33dc6e41a79ee39c420868d49a 2023-04-01 03:57:52,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9390bc8038284059ae9eb4fe1753d8c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/0d3206bcc3f74867895adf9c6df6e4fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/459391686a044c73b22804931ae1b3da, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7b74c5293d10493b90426919de2ac3c5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52] to archive 2023-04-01 03:57:52,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:57:52,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/221aeb5d99ac4ee6a62471b695ab69b0 2023-04-01 03:57:52,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ccdf941b18114ba7b554c3eca08e87d9 2023-04-01 03:57:52,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9390bc8038284059ae9eb4fe1753d8c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9390bc8038284059ae9eb4fe1753d8c0 2023-04-01 03:57:52,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7f1f53982f1d424ab041fcb951022772 2023-04-01 03:57:52,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/b3b8acb0bc5347348baf42beec29545e 2023-04-01 03:57:52,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/0d3206bcc3f74867895adf9c6df6e4fe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/0d3206bcc3f74867895adf9c6df6e4fe 2023-04-01 03:57:52,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6c6937238c2d495c8a07bf9344f3baf3 2023-04-01 03:57:52,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/463d360ae6464c5ca9ab44ee22cbb327 2023-04-01 03:57:52,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/619fe5103da94ea9b475fb1dd46e89e9 2023-04-01 03:57:52,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/40e04bd6f6184a6c939710f2a6318eef 2023-04-01 03:57:52,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1b3149d80a0b4d74bb291c12c65b9316 2023-04-01 03:57:52,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/01121558223e49d9b1b9288dbed062ce 2023-04-01 03:57:52,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/52f08020bf9e48919d396e9631882235 2023-04-01 03:57:52,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/23bcb3cff55b4cceb9fd68168f4e18f1 2023-04-01 03:57:52,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7203f12dec394687894dec9d5ed0ad40 2023-04-01 03:57:52,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/6ff1f365fd28480b8a9f025407d3a7a0 2023-04-01 03:57:52,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/459391686a044c73b22804931ae1b3da to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/459391686a044c73b22804931ae1b3da 2023-04-01 03:57:52,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/716e3d9fdd924f97a9bfbc6783c849b0 2023-04-01 03:57:52,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/1d9e812e18ed4808b91c92d0ac43f5ad 2023-04-01 03:57:52,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/ba3e7c3b2cbe4c15ac7583ff90d35d5d 2023-04-01 03:57:52,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/74cc7d1de550401ebbe510eaf0a0d925 2023-04-01 03:57:52,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/2727ad9dc7fa477781c60b7a256f73a5 2023-04-01 03:57:52,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/457a7647297f46f7b55b107e119dccd7 2023-04-01 03:57:52,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7b74c5293d10493b90426919de2ac3c5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/7b74c5293d10493b90426919de2ac3c5 2023-04-01 03:57:52,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/c81229c498b547f5ade8db10c7bbf804 2023-04-01 03:57:52,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/bda7da5393a34dada1885fa21b266625 2023-04-01 03:57:52,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4829ca5a06434a8e94f196a9ae626574 2023-04-01 03:57:52,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/4a6a1828f27e4f0aba8d93035e2cfa52 2023-04-01 03:57:52,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/recovered.edits/416.seqid, newMaxSeqId=416, maxSeqId=1 2023-04-01 03:57:52,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72. 2023-04-01 03:57:52,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 152d4d46bcbd27b89955640499bc0d72: 2023-04-01 03:57:52,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:52,061 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=152d4d46bcbd27b89955640499bc0d72, regionState=CLOSED 2023-04-01 03:57:52,061 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321472061"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321472061"}]},"ts":"1680321472061"} 2023-04-01 03:57:52,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-01 03:57:52,064 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 152d4d46bcbd27b89955640499bc0d72, server=jenkins-hbase20.apache.org,36251,1680321375816 in 1.1430 sec 2023-04-01 03:57:52,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-01 03:57:52,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=152d4d46bcbd27b89955640499bc0d72, UNASSIGN in 1.1470 sec 2023-04-01 03:57:52,070 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321472070"}]},"ts":"1680321472070"} 2023-04-01 03:57:52,071 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:57:52,074 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:57:52,076 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-01 03:57:53,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 03:57:53,012 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-01 03:57:53,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:57:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,015 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-01 03:57:53,015 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,019 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:53,021 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/recovered.edits] 2023-04-01 03:57:53,024 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/54eeaf9a371e4265ab2e0b72a93ac40b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/54eeaf9a371e4265ab2e0b72a93ac40b 2023-04-01 03:57:53,025 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/986873c3348043788eccf4d94591d0bc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/A/986873c3348043788eccf4d94591d0bc 2023-04-01 03:57:53,027 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/17f1240abcf246aeab5acf24ba93f685 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/17f1240abcf246aeab5acf24ba93f685 2023-04-01 03:57:53,028 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5d5c855d4b44dc5a05fd621565884e4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/B/b5d5c855d4b44dc5a05fd621565884e4 2023-04-01 03:57:53,030 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9a1d022f827e471e901af67c342dcf26 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/9a1d022f827e471e901af67c342dcf26 2023-04-01 03:57:53,031 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/cf73f66b15b94f2e8977c75408b633d3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/C/cf73f66b15b94f2e8977c75408b633d3 2023-04-01 03:57:53,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/recovered.edits/416.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72/recovered.edits/416.seqid 2023-04-01 03:57:53,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/152d4d46bcbd27b89955640499bc0d72 2023-04-01 03:57:53,034 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:57:53,036 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,039 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:57:53,041 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:57:53,042 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,042 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:57:53,042 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321473042"}]},"ts":"9223372036854775807"} 2023-04-01 03:57:53,044 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:57:53,044 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 152d4d46bcbd27b89955640499bc0d72, NAME => 'TestAcidGuarantees,,1680321439703.152d4d46bcbd27b89955640499bc0d72.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:57:53,044 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:57:53,044 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321473044"}]},"ts":"9223372036854775807"} 2023-04-01 03:57:53,047 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:57:53,049 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,050 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-01 03:57:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-01 03:57:53,116 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-01 03:57:53,126 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 229), OpenFileDescriptor=547 (was 553), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=476 (was 364) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=8864 (was 7808) - AvailableMemoryMB LEAK? - 2023-04-01 03:57:53,135 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=476, ProcessCount=169, AvailableMemoryMB=8863 2023-04-01 03:57:53,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:57:53,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:57:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:53,140 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:57:53,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-01 03:57:53,141 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:57:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:53,142 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:53,142 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e empty. 2023-04-01 03:57:53,143 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:53,143 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:57:53,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:53,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:53,554 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:57:53,558 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c34029273b14edd6bcd250b04ada3f0e, NAME => 'TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:57:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c34029273b14edd6bcd250b04ada3f0e, disabling compactions & flushes 2023-04-01 03:57:53,966 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. after waiting 0 ms 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:53,966 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:53,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:53,968 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:57:53,969 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321473969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321473969"}]},"ts":"1680321473969"} 2023-04-01 03:57:53,970 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:57:53,971 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:57:53,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321473971"}]},"ts":"1680321473971"} 2023-04-01 03:57:53,972 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:57:53,974 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, ASSIGN}] 2023-04-01 03:57:53,976 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, ASSIGN 2023-04-01 03:57:53,976 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:57:54,127 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:54,128 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321474127"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321474127"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321474127"}]},"ts":"1680321474127"} 2023-04-01 03:57:54,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:54,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:54,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:54,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c34029273b14edd6bcd250b04ada3f0e, NAME => 'TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:57:54,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:57:54,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,286 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,287 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A 2023-04-01 03:57:54,287 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A 2023-04-01 03:57:54,288 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:54,288 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName A 2023-04-01 03:57:54,288 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:54,288 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,289 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B 2023-04-01 03:57:54,289 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B 2023-04-01 03:57:54,290 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:54,290 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName B 2023-04-01 03:57:54,290 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:54,290 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,291 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C 2023-04-01 03:57:54,291 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C 2023-04-01 03:57:54,291 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:54,292 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName C 2023-04-01 03:57:54,292 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:54,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:54,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:57:54,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:54,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:57:54,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c34029273b14edd6bcd250b04ada3f0e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59513924, jitterRate=-0.11317342519760132}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:57:54,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:54,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., pid=37, masterSystemTime=1680321474280 2023-04-01 03:57:54,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:54,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:54,300 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:54,301 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321474300"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321474300"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321474300"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321474300"}]},"ts":"1680321474300"} 2023-04-01 03:57:54,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-01 03:57:54,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 in 173 msec 2023-04-01 03:57:54,304 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-01 03:57:54,304 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, ASSIGN in 329 msec 2023-04-01 03:57:54,305 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:57:54,305 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321474305"}]},"ts":"1680321474305"} 2023-04-01 03:57:54,306 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:57:54,308 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:57:54,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-01 03:57:55,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 03:57:55,248 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-01 03:57:55,249 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x4dc83d7d to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:55,272 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78e129d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:55,273 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:55,275 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:55,276 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:57:55,277 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:57:55,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:57:55,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:57:55,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 03:57:55,690 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:57:55,691 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:57:55,691 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:57:55,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 03:57:55,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, REOPEN/MOVE}] 2023-04-01 03:57:55,696 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, REOPEN/MOVE 2023-04-01 03:57:55,696 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:55,697 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321475696"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321475696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321475696"}]},"ts":"1680321475696"} 2023-04-01 03:57:55,697 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:55,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:55,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c34029273b14edd6bcd250b04ada3f0e, disabling compactions & flushes 2023-04-01 03:57:55,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:55,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:55,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. after waiting 0 ms 2023-04-01 03:57:55,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:55,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 03:57:55,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:55,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:55,854 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c34029273b14edd6bcd250b04ada3f0e to self. 2023-04-01 03:57:55,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:55,855 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=CLOSED 2023-04-01 03:57:55,856 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321475855"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321475855"}]},"ts":"1680321475855"} 2023-04-01 03:57:55,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-01 03:57:55,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 in 159 msec 2023-04-01 03:57:55,858 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=true 2023-04-01 03:57:56,008 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,008 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321476008"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321476008"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321476008"}]},"ts":"1680321476008"} 2023-04-01 03:57:56,009 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:57:56,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c34029273b14edd6bcd250b04ada3f0e, NAME => 'TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:57:56,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:57:56,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,166 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,166 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A 2023-04-01 03:57:56,166 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A 2023-04-01 03:57:56,167 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:56,167 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName A 2023-04-01 03:57:56,169 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:56,169 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,169 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B 2023-04-01 03:57:56,169 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B 2023-04-01 03:57:56,170 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:56,170 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName B 2023-04-01 03:57:56,170 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:56,170 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,171 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C 2023-04-01 03:57:56,171 DEBUG [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C 2023-04-01 03:57:56,171 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:57:56,171 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c34029273b14edd6bcd250b04ada3f0e columnFamilyName C 2023-04-01 03:57:56,171 INFO [StoreOpener-c34029273b14edd6bcd250b04ada3f0e-1] regionserver.HStore(310): Store=c34029273b14edd6bcd250b04ada3f0e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:57:56,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:57:56,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c34029273b14edd6bcd250b04ada3f0e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70612901, jitterRate=0.05221422016620636}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:57:56,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:56,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., pid=42, masterSystemTime=1680321476161 2023-04-01 03:57:56,180 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321476180"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321476180"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321476180"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321476180"}]},"ts":"1680321476180"} 2023-04-01 03:57:56,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-01 03:57:56,185 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 in 174 msec 2023-04-01 03:57:56,186 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-01 03:57:56,186 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, REOPEN/MOVE in 490 msec 2023-04-01 03:57:56,187 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-01 03:57:56,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 493 msec 2023-04-01 03:57:56,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 909 msec 2023-04-01 03:57:56,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-01 03:57:56,191 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x566231bc to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,208 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5acdcdd8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,208 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x52f2f9ef to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,229 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@155adddc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,229 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x142c61b8 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,237 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5154cf53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,237 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x58a97f1f to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,241 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f7fe413, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,241 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x50e948f8 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,244 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@432b847d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,244 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x651d3d36 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,247 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a0a5584, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,247 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x7bd89b96 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,250 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77a0ed56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,250 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x0f0877a9 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,255 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30b359f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,256 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6e0b4412 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,260 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22f2d345, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,261 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x06e48d7b to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:57:56,265 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@327b6261, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:57:56,272 DEBUG [hconnection-0x3627dfcc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,272 DEBUG [hconnection-0x738e52c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,272 DEBUG [hconnection-0x361d850e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,273 DEBUG [hconnection-0x6cc895b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:56,275 DEBUG [hconnection-0x420def6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,275 DEBUG [hconnection-0x75e2bcbb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,275 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,275 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41908, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,277 DEBUG [hconnection-0x57eceb20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,277 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,275 DEBUG [hconnection-0xa6516ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,279 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,279 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,278 DEBUG [hconnection-0x547e092b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:56,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:56,284 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:56,284 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,285 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:56,285 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:56,285 DEBUG [hconnection-0x4167b845-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:57:56,285 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,285 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,287 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,287 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,287 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:57:56,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 03:57:56,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:56,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:57:56,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:56,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:56,308 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,308 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:56,308 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:56,308 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:56,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:56,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,311 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,312 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,312 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79226857[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:56,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:56,313 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:56,314 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:56,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:56,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:56,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:56,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,315 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:56,315 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:56,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:56,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:56,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:56,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:56,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:56,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:56,319 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,319 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,319 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,322 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,325 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:56,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:56,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321536327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321536328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321536328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,330 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,330 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,334 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:56,334 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:56,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321536334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321536336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,354 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401a08f6897812b46ccaf075b3b66bdd4f0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a08f6897812b46ccaf075b3b66bdd4f0 2023-04-01 03:57:56,378 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/7aad55ec3c59422892bac628e01bdd97 2023-04-01 03:57:56,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321536432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,435 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:56,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321536435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321536435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321536437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321536437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:56,441 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:56,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:56,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:56,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,443 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:56,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:56,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:56,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:56,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:56,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:56,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:56,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:56,449 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,449 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:56,449 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:56,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:56,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:56,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:56,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,470 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:56,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d1e16d0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:56,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:56,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:56,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:56,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:56,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:56,476 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:56,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:56,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:56,476 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:56,476 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:56,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,479 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:56,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:56,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,503 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,503 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:56,504 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:56,503 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,504 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,604 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:56,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:56,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:56,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:56,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:56,606 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:56,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:56,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:56,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:56,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:56,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:56,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:56,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:56,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:56,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:56,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:56,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:56,611 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:56,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:56,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:56,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a1b283[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:56,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,614 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:56,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:56,614 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:56,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:56,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:56,617 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:56,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:56,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:56,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:56,618 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:56,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,621 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:56,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:56,622 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,623 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,623 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:56,623 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:56,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321536633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321536637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321536638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321536639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321536639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,723 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:56,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:56,731 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:56,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:56,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:56,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:56,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:56,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:56,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:56,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e45d43b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:56,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:56,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,758 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:56,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:56,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:56,759 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:56,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:56,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:56,762 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:56,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:56,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:56,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:56,762 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:56,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:56,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:56,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:56,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:56,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:56,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,771 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:56,771 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:56,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:56,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/fbd34abc2a5745488b3507438de19011 2023-04-01 03:57:56,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/b15e8f5e53404a7b9c6429f0c8ecce75 2023-04-01 03:57:56,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/7aad55ec3c59422892bac628e01bdd97 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97 2023-04-01 03:57:56,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97, entries=250, sequenceid=18, filesize=38.7 K 2023-04-01 03:57:56,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/fbd34abc2a5745488b3507438de19011 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011 2023-04-01 03:57:56,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011, entries=150, sequenceid=18, filesize=11.6 K 2023-04-01 03:57:56,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/b15e8f5e53404a7b9c6429f0c8ecce75 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75 2023-04-01 03:57:56,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75, entries=150, sequenceid=18, filesize=11.6 K 2023-04-01 03:57:56,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for c34029273b14edd6bcd250b04ada3f0e in 537ms, sequenceid=18, compaction requested=false 2023-04-01 03:57:56,840 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-01 03:57:56,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:56,871 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:56,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:56,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:56,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:56,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:56,878 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:56,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:56,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:56,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,880 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:56,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:56,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:56,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:56,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:56,884 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:57:56,884 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:56,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:56,946 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:56,957 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401f73429369696419d9a6ab38a2f637504 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f73429369696419d9a6ab38a2f637504 2023-04-01 03:57:56,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321536955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321536956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321536957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321536958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321536959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:56,989 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/25847fe9d1a64dd6a965bff0c4f902e8 2023-04-01 03:57:57,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321537061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321537065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321537066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321537067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321537068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321537270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321537271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321537271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321537273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321537282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,400 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/34b036c0309d434d95319036bc7b9470 2023-04-01 03:57:57,408 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/3e1edd0c997c4837bd066c6f3dedbeb1 2023-04-01 03:57:57,411 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/25847fe9d1a64dd6a965bff0c4f902e8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8 2023-04-01 03:57:57,414 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8, entries=150, sequenceid=37, filesize=25.1 K 2023-04-01 03:57:57,414 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/34b036c0309d434d95319036bc7b9470 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470 2023-04-01 03:57:57,417 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 03:57:57,418 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/3e1edd0c997c4837bd066c6f3dedbeb1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1 2023-04-01 03:57:57,421 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 03:57:57,421 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c34029273b14edd6bcd250b04ada3f0e in 537ms, sequenceid=37, compaction requested=false 2023-04-01 03:57:57,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:57,421 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:57,421 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:57,423 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:57,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:57,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:57,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14cf7b74[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:57,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:57,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:57,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:57,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:57,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:57,428 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:57,432 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:57,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:57,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:57,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:57,432 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:57,434 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:57,434 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,434 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:57,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:57,437 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:57,437 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,437 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:57,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:57,438 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:57,538 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:57,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:57,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:57,544 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:57,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:57,547 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:57,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:57,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:57,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:57,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:57,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:57,551 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:57:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:57,575 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018ba167ee19264afbbdfbedaeec9ee304 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018ba167ee19264afbbdfbedaeec9ee304 2023-04-01 03:57:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:57,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:57,591 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/668378d789d54d7f9cca5f9d6a3ea0f1 2023-04-01 03:57:57,591 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:57:57,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321537606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321537607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321537607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321537615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321537617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321537718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321537718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321537718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321537719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321537725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321537926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321537926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321537926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321537927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:57,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321537928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,033 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/282fb8b9f4134f68969833e7d59f9599 2023-04-01 03:57:58,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321538232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321538233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321538233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321538233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321538234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,444 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/445d7900182a4c77980fc9521a1c2eb8 2023-04-01 03:57:58,449 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/668378d789d54d7f9cca5f9d6a3ea0f1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1 2023-04-01 03:57:58,453 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1, entries=150, sequenceid=50, filesize=25.1 K 2023-04-01 03:57:58,454 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/282fb8b9f4134f68969833e7d59f9599 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599 2023-04-01 03:57:58,460 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599, entries=150, sequenceid=50, filesize=11.6 K 2023-04-01 03:57:58,461 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/445d7900182a4c77980fc9521a1c2eb8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8 2023-04-01 03:57:58,466 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8, entries=150, sequenceid=50, filesize=11.6 K 2023-04-01 03:57:58,467 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c34029273b14edd6bcd250b04ada3f0e in 916ms, sequenceid=50, compaction requested=true 2023-04-01 03:57:58,467 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:57:58,467 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:57:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:57:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:57:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:57:58,470 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:58,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:57:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:58,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:57:58,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9fa47ab[Count = 0] remaining members to acquire global barrier 2023-04-01 03:57:58,472 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:57:58,472 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,473 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:57:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:57:58,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:57:58,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:57:58,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:57:58,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:58,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:57:58,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:57:58,478 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:57:58,478 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:57:58,478 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:57:58,478 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:57:58,479 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:57:58,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:57:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:57:58,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:57:58,496 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:57:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:57:58,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:57:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:57:58,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:57:58,505 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:57:58,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:57:58,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:57:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:57:58,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:57:58,506 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:57:58,506 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:57:58,606 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:57:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:57:58,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:57:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:57:58,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:57:58,608 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:57:58,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:58,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:57:58,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:57:58,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:58,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,611 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:57:58,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:57:58,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:57:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:58,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:57:58,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:57:58,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:57:58,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:57:58,612 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:57:58,612 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:57:58,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:57:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:57:58,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:57:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321538746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321538752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321538753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321538753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321538753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321538854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321538855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321538859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321538859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321538860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,041 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ba23674eae024c5e87716cc6bf298b24 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba23674eae024c5e87716cc6bf298b24 2023-04-01 03:57:59,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321539055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321539067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321539067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321539068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321539068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321539360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321539374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321539375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321539375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321539376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,445 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/d29209908f3b4465807b2c7e569c0286 2023-04-01 03:57:59,864 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/f00c1d9f32f24498a3b95984497141db 2023-04-01 03:57:59,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321539864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321539881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321539882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321539885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:57:59,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:57:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321539885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,276 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/0be65dd9e7b1471ab56ce7bc42505f2c 2023-04-01 03:58:00,279 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/d29209908f3b4465807b2c7e569c0286 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286 2023-04-01 03:58:00,282 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286, entries=150, sequenceid=69, filesize=25.1 K 2023-04-01 03:58:00,283 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/f00c1d9f32f24498a3b95984497141db as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db 2023-04-01 03:58:00,286 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db, entries=150, sequenceid=69, filesize=11.6 K 2023-04-01 03:58:00,286 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/0be65dd9e7b1471ab56ce7bc42505f2c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c 2023-04-01 03:58:00,294 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c, entries=150, sequenceid=69, filesize=11.6 K 2023-04-01 03:58:00,295 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c34029273b14edd6bcd250b04ada3f0e in 1683ms, sequenceid=69, compaction requested=true 2023-04-01 03:58:00,295 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:00,295 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:00,297 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,297 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:00,297 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,297 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:00,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:00,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:00,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6224c9bd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:00,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,300 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:00,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:00,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:00,300 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,300 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:00,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:00,301 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:00,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:00,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:00,303 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:00,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:00,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:00,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:00,303 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:00,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:00,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:00,325 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:00,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:00,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:00,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:00,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:00,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:00,328 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:00,329 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:00,329 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:00,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:00,429 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:00,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:00,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:00,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:00,432 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:00,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:00,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:00,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:00,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:00,433 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:00,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:00,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:00,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:00,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:00,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:00,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:00,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:00,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:00,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:00,435 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:58:00,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:00,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:00,435 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:00,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:00,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:00,436 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:00,463 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011e5d4c05f6eb418a93e3b6c60467d020 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011e5d4c05f6eb418a93e3b6c60467d020 2023-04-01 03:58:00,873 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c7394271dd7240ecb97598391cea51fa 2023-04-01 03:58:00,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:00,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321540916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321540917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321540924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321540925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:00,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321540927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321541030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321541030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321541039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321541039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321541039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321541235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321541235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321541246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321541246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321541247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,308 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/7efcc0541003476982f80e5a02f67e71 2023-04-01 03:58:01,373 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/36b05261ef23447a86cad0fa31418e0a 2023-04-01 03:58:01,379 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c7394271dd7240ecb97598391cea51fa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa 2023-04-01 03:58:01,384 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa, entries=150, sequenceid=83, filesize=25.1 K 2023-04-01 03:58:01,386 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/7efcc0541003476982f80e5a02f67e71 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71 2023-04-01 03:58:01,420 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71, entries=150, sequenceid=83, filesize=11.6 K 2023-04-01 03:58:01,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/36b05261ef23447a86cad0fa31418e0a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a 2023-04-01 03:58:01,439 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a, entries=150, sequenceid=83, filesize=11.6 K 2023-04-01 03:58:01,439 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c34029273b14edd6bcd250b04ada3f0e in 1004ms, sequenceid=83, compaction requested=true 2023-04-01 03:58:01,440 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:01,440 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:01,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:01,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:01,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:01,447 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,447 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:01,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32102dce[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:01,451 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:01,451 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:01,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:01,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,463 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:01,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:01,466 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:01,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:01,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:01,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:01,466 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:01,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,493 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:01,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:01,500 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:01,500 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:01,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,499 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,500 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,500 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,500 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,500 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:01,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:58:01,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:01,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:01,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:01,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:01,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:01,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:01,599 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040154add8aec618401780f9cd86af34ae54 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040154add8aec618401780f9cd86af34ae54 2023-04-01 03:58:01,600 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:01,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:01,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:01,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321541603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321541607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321541607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321541608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321541610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:01,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:01,622 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:01,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:01,622 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:01,623 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,624 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,626 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,626 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,628 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:01,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:01,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:01,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:01,638 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,638 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:01,638 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:01,638 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:01,638 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/30413ca1ed194239bc758e6afdb9d64c 2023-04-01 03:58:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:01,645 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,645 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,645 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,645 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:01,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6bf423[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:01,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:01,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:01,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:01,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:01,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,650 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,650 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:01,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:01,652 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:01,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:01,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:01,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:01,652 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:01,653 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,655 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,655 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:01,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,656 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,656 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,656 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:01,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321541711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321541715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321541716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321541716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321541716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,756 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:01,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:01,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:01,759 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:01,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:01,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:01,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,762 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:01,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:01,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:01,763 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,763 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:01,763 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:01,763 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:01,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:01,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:01,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7373a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:01,768 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:01,768 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,771 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:01,771 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:01,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:01,771 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:01,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:01,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:01,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:01,776 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:01,777 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:01,777 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:01,777 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:01,777 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:01,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,777 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,778 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,779 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,780 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,781 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,781 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:01,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,782 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:01,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:01,782 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:01,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,782 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,882 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:01,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:01,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:01,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:01,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:01,884 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:01,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:01,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:01,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,887 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,887 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,887 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,887 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:01,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:01,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:01,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:01,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:01,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:01,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:01,888 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,888 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:01,888 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:01,889 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:01,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:01,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:01,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:01,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:01,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:01,892 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42da6f78[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:01,892 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,894 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,894 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:01,894 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:01,894 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,894 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:01,894 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:01,896 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:01,896 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,896 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,896 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:01,898 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:01,898 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:01,898 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:01,898 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:01,898 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:01,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:01,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:01,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:01,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:01,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:01,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:01,903 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:01,904 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:01,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:01,904 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:01,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:01,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321541917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321541920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321541920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321541921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321541921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,004 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:02,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:02,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:02,007 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:02,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:02,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:02,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:02,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:02,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:02,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:02,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:02,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:02,009 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,009 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:02,009 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:02,009 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:02,009 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:02,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:02,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:02,010 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:02,011 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,011 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:02,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:02,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@719754b0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:02,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:02,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:02,012 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:02,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:02,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:02,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:02,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:02,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:02,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:02,015 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:02,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:02,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:02,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:02,015 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:02,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:02,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:02,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:02,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:02,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:02,019 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:02,019 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,019 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:02,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/ac246290fd9b46e2aa75c14107fb905b 2023-04-01 03:58:02,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/8394761f986241bdaeac8c146c7028d1 2023-04-01 03:58:02,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/30413ca1ed194239bc758e6afdb9d64c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c 2023-04-01 03:58:02,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c, entries=150, sequenceid=102, filesize=25.1 K 2023-04-01 03:58:02,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/ac246290fd9b46e2aa75c14107fb905b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b 2023-04-01 03:58:02,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b, entries=100, sequenceid=102, filesize=9.3 K 2023-04-01 03:58:02,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/8394761f986241bdaeac8c146c7028d1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1 2023-04-01 03:58:02,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1, entries=100, sequenceid=102, filesize=9.3 K 2023-04-01 03:58:02,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c34029273b14edd6bcd250b04ada3f0e in 522ms, sequenceid=102, compaction requested=true 2023-04-01 03:58:02,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:02,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:02,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:02,096 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:58:02,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:02,096 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:58:02,097 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68972 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:58:02,097 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:02,097 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,097 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=67.4 K 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fbd34abc2a5745488b3507438de19011, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680321476300 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168356 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:02,098 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,098 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=164.4 K 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 34b036c0309d434d95319036bc7b9470, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680321476313 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7aad55ec3c59422892bac628e01bdd97, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680321476300 2023-04-01 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 282fb8b9f4134f68969833e7d59f9599, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680321476953 2023-04-01 03:58:02,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 25847fe9d1a64dd6a965bff0c4f902e8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680321476313 2023-04-01 03:58:02,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f00c1d9f32f24498a3b95984497141db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680321477589 2023-04-01 03:58:02,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 668378d789d54d7f9cca5f9d6a3ea0f1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680321476953 2023-04-01 03:58:02,099 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d29209908f3b4465807b2c7e569c0286, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680321477589 2023-04-01 03:58:02,099 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7efcc0541003476982f80e5a02f67e71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321478743 2023-04-01 03:58:02,101 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c7394271dd7240ecb97598391cea51fa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321478743 2023-04-01 03:58:02,101 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ac246290fd9b46e2aa75c14107fb905b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321480925 2023-04-01 03:58:02,101 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 30413ca1ed194239bc758e6afdb9d64c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321480922 2023-04-01 03:58:02,112 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#306 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:02,119 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:02,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:02,134 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:02,134 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:02,134 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:02,134 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:02,134 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,134 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,136 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:02,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:02,136 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,136 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:02,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:02,145 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#307 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:02,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:02,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:02,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:02,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:02,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:02,150 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:02,150 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:02,150 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,150 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:02,150 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:02,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/b38e69387d1d4129ad0fd861f59f442d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b38e69387d1d4129ad0fd861f59f442d 2023-04-01 03:58:02,163 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into b38e69387d1d4129ad0fd861f59f442d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:02,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:02,163 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=10, startTime=1680321482096; duration=0sec 2023-04-01 03:58:02,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:02,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:58:02,165 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68972 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:58:02,165 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:02,165 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:02,165 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=67.4 K 2023-04-01 03:58:02,165 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b15e8f5e53404a7b9c6429f0c8ecce75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680321476300 2023-04-01 03:58:02,165 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 3e1edd0c997c4837bd066c6f3dedbeb1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680321476313 2023-04-01 03:58:02,166 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 445d7900182a4c77980fc9521a1c2eb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680321476953 2023-04-01 03:58:02,166 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0be65dd9e7b1471ab56ce7bc42505f2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680321477589 2023-04-01 03:58:02,166 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 36b05261ef23447a86cad0fa31418e0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321478743 2023-04-01 03:58:02,167 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8394761f986241bdaeac8c146c7028d1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321480925 2023-04-01 03:58:02,181 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#308 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:02,192 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401351c40484a1f4a9c90c6eb8dd8e2eb39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401351c40484a1f4a9c90c6eb8dd8e2eb39 2023-04-01 03:58:02,192 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/6881efc053c1454c83308866122e13e3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/6881efc053c1454c83308866122e13e3 2023-04-01 03:58:02,196 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into 6881efc053c1454c83308866122e13e3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:02,196 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:02,196 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=10, startTime=1680321482096; duration=0sec 2023-04-01 03:58:02,196 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:02,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:02,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321542244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321542250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321542250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321542251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321542251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321542352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321542357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321542357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321542358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321542358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321542555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321542564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321542565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321542565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321542568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,600 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c54c992df0ba49a1bf834c0de6627f51 2023-04-01 03:58:02,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321542859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321542871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321542873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321542873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:02,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:02,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321542874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,011 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/c751a53063e9417fb8a1970386e86082 2023-04-01 03:58:03,026 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/66b32c3126064207a4c8dd57d40cdac3 2023-04-01 03:58:03,030 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c54c992df0ba49a1bf834c0de6627f51 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51 2023-04-01 03:58:03,033 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51, entries=150, sequenceid=115, filesize=25.1 K 2023-04-01 03:58:03,033 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/c751a53063e9417fb8a1970386e86082 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082 2023-04-01 03:58:03,036 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082, entries=150, sequenceid=115, filesize=11.6 K 2023-04-01 03:58:03,037 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/66b32c3126064207a4c8dd57d40cdac3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3 2023-04-01 03:58:03,039 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3, entries=150, sequenceid=115, filesize=11.6 K 2023-04-01 03:58:03,040 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for c34029273b14edd6bcd250b04ada3f0e in 890ms, sequenceid=115, compaction requested=false 2023-04-01 03:58:03,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:03,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:03,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:03,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:03,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:03,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:03,042 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:03,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73df6c4d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:03,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:03,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,044 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:03,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:03,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:03,045 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:03,046 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:03,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:03,048 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:03,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:03,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:03,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:03,048 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:03,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:03,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:03,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,070 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:03,073 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:03,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:03,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,073 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:03,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:03,073 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:03,174 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:03,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:03,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:03,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:03,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:03,176 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:03,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:03,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:03,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:03,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:03,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:03,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:03,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:03,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:03,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:03,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:03,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:03,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:03,179 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:03,179 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:03,179 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:03,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:03,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:03,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321543383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321543385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321543388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,399 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/2c64965c8cb04826a3d8d1c96284be0d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/2c64965c8cb04826a3d8d1c96284be0d 2023-04-01 03:58:03,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321543389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321543389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,409 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into 2c64965c8cb04826a3d8d1c96284be0d(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:58:03,409 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:03,409 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=10, startTime=1680321482096; duration=1sec 2023-04-01 03:58:03,409 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:03,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321543490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321543497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321543498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321543501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321543501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,599 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304013a2bb2cb561d43c9bb59b286d0ea8414 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013a2bb2cb561d43c9bb59b286d0ea8414 2023-04-01 03:58:03,626 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/9429f047e0ef4ff7be0f9b62af4544d3 2023-04-01 03:58:03,692 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/0207b00376844b9292440ae0d93febb4 2023-04-01 03:58:03,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321543694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321543701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321543702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321543707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:03,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321543708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,767 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5c6f3cb33546465a8df9af232e7e2927 2023-04-01 03:58:03,774 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/9429f047e0ef4ff7be0f9b62af4544d3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3 2023-04-01 03:58:03,781 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3, entries=150, sequenceid=137, filesize=25.3 K 2023-04-01 03:58:03,782 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/0207b00376844b9292440ae0d93febb4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4 2023-04-01 03:58:03,818 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4, entries=150, sequenceid=137, filesize=11.8 K 2023-04-01 03:58:03,819 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5c6f3cb33546465a8df9af232e7e2927 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927 2023-04-01 03:58:03,823 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927, entries=150, sequenceid=137, filesize=11.8 K 2023-04-01 03:58:03,824 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for c34029273b14edd6bcd250b04ada3f0e in 645ms, sequenceid=137, compaction requested=true 2023-04-01 03:58:03,824 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:03,824 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:03,824 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:03,827 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:03,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4caecbeb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:03,829 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:03,829 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,830 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,830 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:03,833 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:03,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:03,833 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:03,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:03,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:03,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:03,850 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:03,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:03,853 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:03,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:03,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:03,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:03,853 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:03,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:03,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:03,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,872 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:03,901 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:03,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:03,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:03,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:03,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:03,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:03,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:03,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:03,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:03,901 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:03,901 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,001 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:04,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,018 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,020 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,040 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321544042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321544042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321544043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321544044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d1ecca9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,056 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012eb0709485344995a4e8c8b867cc541f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb0709485344995a4e8c8b867cc541f 2023-04-01 03:58:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,058 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321544052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,064 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,064 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,070 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,070 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,070 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,071 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,073 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,074 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,074 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,074 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321544153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321544153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321544158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321544159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321544163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,177 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,181 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,182 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,182 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,183 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,198 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,198 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1667216e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,203 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,203 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,204 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,204 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,205 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,206 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,218 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,218 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,218 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,218 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,218 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,221 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,223 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,230 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,230 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,230 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,232 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,232 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,332 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,338 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,338 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,338 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,338 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,361 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,364 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321544361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e529aeb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,364 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321544361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,370 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,372 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,372 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321544365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321544366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,376 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,376 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321544374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,382 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,382 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:04,481 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/ac8002d897ca42daaa37307f30536d59 2023-04-01 03:58:04,482 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,489 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,492 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,492 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,508 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,508 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,522 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,524 WARN [DataStreamer for file /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/65db88baf7b34e64b9be621bc4e233dc] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 03:58:04,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/65db88baf7b34e64b9be621bc4e233dc 2023-04-01 03:58:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2499eb59[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,526 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,526 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,529 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,534 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,535 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,542 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,543 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/f9e969ec4d65410b8c935ecb17daad39 2023-04-01 03:58:04,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/ac8002d897ca42daaa37307f30536d59 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59 2023-04-01 03:58:04,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59, entries=200, sequenceid=152, filesize=32.1 K 2023-04-01 03:58:04,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/65db88baf7b34e64b9be621bc4e233dc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc 2023-04-01 03:58:04,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc, entries=150, sequenceid=152, filesize=11.8 K 2023-04-01 03:58:04,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/f9e969ec4d65410b8c935ecb17daad39 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39 2023-04-01 03:58:04,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39, entries=150, sequenceid=152, filesize=11.8 K 2023-04-01 03:58:04,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c34029273b14edd6bcd250b04ada3f0e in 620ms, sequenceid=152, compaction requested=true 2023-04-01 03:58:04,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:04,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:04,626 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:04,626 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:04,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:04,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:04,627 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110480 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:04,628 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:04,628 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,628 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/2c64965c8cb04826a3d8d1c96284be0d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=107.9 K 2023-04-01 03:58:04,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:04,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:04,628 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,628 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b38e69387d1d4129ad0fd861f59f442d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=46.9 K 2023-04-01 03:58:04,628 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2c64965c8cb04826a3d8d1c96284be0d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321478743 2023-04-01 03:58:04,628 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b38e69387d1d4129ad0fd861f59f442d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321478743 2023-04-01 03:58:04,629 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c54c992df0ba49a1bf834c0de6627f51, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680321481605 2023-04-01 03:58:04,629 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c751a53063e9417fb8a1970386e86082, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680321481605 2023-04-01 03:58:04,629 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9429f047e0ef4ff7be0f9b62af4544d3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680321482236 2023-04-01 03:58:04,629 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0207b00376844b9292440ae0d93febb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680321482236 2023-04-01 03:58:04,630 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ac8002d897ca42daaa37307f30536d59, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:04,630 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 65db88baf7b34e64b9be621bc4e233dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:04,643 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,652 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#319 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:04,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:04,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:04,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:04,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,677 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,679 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,698 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,698 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,698 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,698 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,698 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,698 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,698 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,698 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,698 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,717 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321544705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f4900e7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321544706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321544705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,722 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,726 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,726 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,728 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,728 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,728 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,731 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,733 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,734 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,734 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,736 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#318 average throughput is 0.20 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:04,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321544720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321544720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321544821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321544821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321544821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,830 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304014723f65d981f482d8b7cadf6d10ded8f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014723f65d981f482d8b7cadf6d10ded8f 2023-04-01 03:58:04,834 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,838 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,838 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,838 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,838 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,843 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,843 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321544837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321544838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,847 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,848 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,848 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,849 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,849 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,849 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,851 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200a3c31[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,854 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,855 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,857 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,857 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:04,859 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,859 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,859 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:04,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:04,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:04,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:04,865 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:04,865 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:04,965 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:04,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:04,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:04,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:04,972 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:04,972 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,972 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:04,972 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:04,972 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,972 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,974 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,974 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:04,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:04,974 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,974 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:04,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:04,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:04,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:04,977 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:04,977 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:04,977 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:04,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:04,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:04,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:04,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:04,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:04,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:04,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:04,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:04,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:04,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:04,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a857f19[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:04,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,994 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:04,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:04,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:04,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:04,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:04,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,995 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:04,995 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:04,995 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:04,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:04,998 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:04,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:04,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:04,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:04,998 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,025 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,025 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,025 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,028 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321545026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321545026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321545026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,050 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,050 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,050 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:05,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321545047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321545047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,125 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/391cc9cd27364a39af5e28fb0523f87b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/391cc9cd27364a39af5e28fb0523f87b 2023-04-01 03:58:05,129 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into 391cc9cd27364a39af5e28fb0523f87b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:05,129 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:05,129 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=12, startTime=1680321484626; duration=0sec 2023-04-01 03:58:05,130 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:05,130 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:05,131 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:05,131 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:05,131 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,131 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/6881efc053c1454c83308866122e13e3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=46.9 K 2023-04-01 03:58:05,131 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6881efc053c1454c83308866122e13e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680321478743 2023-04-01 03:58:05,131 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 66b32c3126064207a4c8dd57d40cdac3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680321481605 2023-04-01 03:58:05,132 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 5c6f3cb33546465a8df9af232e7e2927, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680321482236 2023-04-01 03:58:05,132 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f9e969ec4d65410b8c935ecb17daad39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:05,147 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:05,151 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,154 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,155 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,155 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412d9a33[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,174 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,174 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,176 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,177 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,177 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,177 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,186 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,186 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,190 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,190 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,191 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,197 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,198 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,198 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,244 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/f730155171f842f9a4d281a7dab15617 2023-04-01 03:58:05,277 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/50b16efdbfc74db6bff9d85aa73ff809 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/50b16efdbfc74db6bff9d85aa73ff809 2023-04-01 03:58:05,291 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into 50b16efdbfc74db6bff9d85aa73ff809(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:05,291 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:05,292 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=12, startTime=1680321484626; duration=0sec 2023-04-01 03:58:05,292 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:05,298 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,302 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,303 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,314 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624e4c9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,322 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,322 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,322 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,324 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,324 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,324 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,324 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,330 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,330 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,330 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,330 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,330 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,336 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,337 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,337 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,337 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,337 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:05,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321545342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321545342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321545344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321545353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321545355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,437 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,442 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,454 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,454 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,454 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,454 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,454 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,456 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bebb88[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,459 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,463 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,470 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,470 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,479 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,483 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,483 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,483 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,565 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/c42568f75dc64756a75134ea8af4cedb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c42568f75dc64756a75134ea8af4cedb 2023-04-01 03:58:05,571 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into c42568f75dc64756a75134ea8af4cedb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:05,571 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:05,571 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=12, startTime=1680321484626; duration=0sec 2023-04-01 03:58:05,571 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:05,583 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,594 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,602 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,602 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,602 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,602 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,604 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,604 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,605 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c27fd1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,605 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,609 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,611 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,611 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,615 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,620 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,620 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,620 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,620 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/1f0fe36a98734b8998babce102f63658 2023-04-01 03:58:05,720 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/719ca80c55e14a92ba12c9926a48ef92 2023-04-01 03:58:05,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,729 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,730 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/f730155171f842f9a4d281a7dab15617 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617 2023-04-01 03:58:05,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,736 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617, entries=200, sequenceid=171, filesize=32.1 K 2023-04-01 03:58:05,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/1f0fe36a98734b8998babce102f63658 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658 2023-04-01 03:58:05,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,742 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b138f38[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,748 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,748 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,749 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,754 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,759 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,759 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,759 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658, entries=150, sequenceid=171, filesize=11.8 K 2023-04-01 03:58:05,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,763 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/719ca80c55e14a92ba12c9926a48ef92 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92 2023-04-01 03:58:05,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,768 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,768 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92, entries=150, sequenceid=171, filesize=11.8 K 2023-04-01 03:58:05,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c34029273b14edd6bcd250b04ada3f0e in 1100ms, sequenceid=171, compaction requested=false 2023-04-01 03:58:05,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:05,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:05,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:05,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:05,868 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:05,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:05,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:05,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:05,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:05,879 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:05,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:05,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:05,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,880 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:05,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:05,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:05,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:05,890 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:05,890 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:05,890 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:05,890 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:05,890 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:05,890 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,891 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:05,891 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:05,891 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:05,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:05,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:05,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:05,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:05,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321545880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321545882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321545883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321545886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,897 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012d589edb5cc0493c9b9b5fefdbcb7e27 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012d589edb5cc0493c9b9b5fefdbcb7e27 2023-04-01 03:58:05,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:05,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321545886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,909 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:05,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a302a3f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:05,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:05,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:05,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:05,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:05,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:05,911 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,911 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=188, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/ab166906ed1849d1994ab02819af6b18 2023-04-01 03:58:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:05,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:05,914 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:05,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:05,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:05,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:05,914 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:05,915 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,915 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:05,915 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:05,916 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,919 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:05,919 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:05,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:05,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:05,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:05,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:05,920 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:05,920 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:05,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:05,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:05,922 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321545994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321545994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321545994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321545997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321545999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,020 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:06,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:06,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:06,026 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:06,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:06,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:06,027 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,027 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,028 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,028 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:06,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:06,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:06,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:06,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:06,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:06,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:06,030 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,030 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:06,030 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:06,030 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,031 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:06,031 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:06,031 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,031 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:06,031 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:06,032 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,032 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,032 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:06,032 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:06,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d0bc0a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:06,033 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:06,033 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,034 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,034 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:06,034 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:06,034 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:06,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:06,035 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:06,035 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,035 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:06,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:06,047 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:06,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:06,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:06,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:06,047 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:06,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,050 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:06,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:06,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,061 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,061 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:06,061 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,161 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:06,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:06,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:06,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:06,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:06,166 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:06,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:06,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:06,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,168 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,168 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,169 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,169 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:06,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:06,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:06,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:06,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:06,178 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:06,178 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:06,178 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,178 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:06,178 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:06,178 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:06,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:06,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:06,179 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,181 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:06,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d896836[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:06,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:06,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,186 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:06,187 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,187 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:06,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:06,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:06,190 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:06,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:06,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:06,190 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:06,190 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:06,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:06,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:06,198 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:06,198 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,202 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,202 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:06,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321546204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321546204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321546204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321546207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321546210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,299 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:06,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:06,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:06,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:06,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:06,302 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:06,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:06,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:06,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:06,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:06,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:06,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:06,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:06,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:06,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,308 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:06,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a245ef0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:06,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:06,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,311 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:06,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,311 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:06,312 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:06,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:06,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:06,316 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:06,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:06,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:06,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:06,317 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:06,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,318 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,318 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,320 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,323 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:06,322 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:06,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:06,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,323 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:06,323 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:06,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:06,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:06,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/c156402bf9614d1b984946bcb6f69433 2023-04-01 03:58:06,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/e70aa956a2fc45399d9a8c11a8d52a20 2023-04-01 03:58:06,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/ab166906ed1849d1994ab02819af6b18 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18 2023-04-01 03:58:06,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18, entries=200, sequenceid=188, filesize=32.1 K 2023-04-01 03:58:06,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/c156402bf9614d1b984946bcb6f69433 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433 2023-04-01 03:58:06,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433, entries=150, sequenceid=188, filesize=11.8 K 2023-04-01 03:58:06,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/e70aa956a2fc45399d9a8c11a8d52a20 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20 2023-04-01 03:58:06,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20, entries=150, sequenceid=188, filesize=11.8 K 2023-04-01 03:58:06,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c34029273b14edd6bcd250b04ada3f0e in 532ms, sequenceid=188, compaction requested=true 2023-04-01 03:58:06,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:06,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:06,384 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:06,384 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:06,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:06,385 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:06,386 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92010 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:06,386 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:06,386 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,386 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/50b16efdbfc74db6bff9d85aa73ff809, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=89.9 K 2023-04-01 03:58:06,386 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:06,386 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:06,386 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,386 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/391cc9cd27364a39af5e28fb0523f87b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.6 K 2023-04-01 03:58:06,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 50b16efdbfc74db6bff9d85aa73ff809, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:06,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f730155171f842f9a4d281a7dab15617, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680321484040 2023-04-01 03:58:06,387 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 391cc9cd27364a39af5e28fb0523f87b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:06,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ab166906ed1849d1994ab02819af6b18, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:06,387 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1f0fe36a98734b8998babce102f63658, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680321484040 2023-04-01 03:58:06,388 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c156402bf9614d1b984946bcb6f69433, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:06,399 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#328 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:06,415 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#327 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:06,429 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:06,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:06,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:06,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:06,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:06,433 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:06,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:06,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:06,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,435 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:06,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:06,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:06,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:06,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:06,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:06,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:06,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:06,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:06,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:06,437 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,437 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:06,438 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:06,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:06,468 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401a07ad9c35e854febbb7a7eec426ae58d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a07ad9c35e854febbb7a7eec426ae58d 2023-04-01 03:58:06,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:06,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:06,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1680321546522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321546523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321546523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321546526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321546526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321546632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321546632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321546632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321546632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,812 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/0caa94c4050d404485ce8e1f44cd94fd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0caa94c4050d404485ce8e1f44cd94fd 2023-04-01 03:58:06,815 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into 0caa94c4050d404485ce8e1f44cd94fd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:06,815 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:06,815 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=13, startTime=1680321486384; duration=0sec 2023-04-01 03:58:06,815 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:06,815 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:06,816 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:06,816 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:06,816 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,816 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c42568f75dc64756a75134ea8af4cedb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.6 K 2023-04-01 03:58:06,817 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c42568f75dc64756a75134ea8af4cedb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680321483373 2023-04-01 03:58:06,817 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 719ca80c55e14a92ba12c9926a48ef92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680321484040 2023-04-01 03:58:06,817 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e70aa956a2fc45399d9a8c11a8d52a20, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:06,821 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:06,835 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/746c04329c4a4ab2b07646999bc6e244 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/746c04329c4a4ab2b07646999bc6e244 2023-04-01 03:58:06,838 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into 746c04329c4a4ab2b07646999bc6e244(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:06,838 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:06,839 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=13, startTime=1680321486385; duration=0sec 2023-04-01 03:58:06,839 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:06,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321546840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321546840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321546840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321546841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,863 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/0bdd5c51c84544f999f2454750972d34 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/0bdd5c51c84544f999f2454750972d34 2023-04-01 03:58:06,869 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into 0bdd5c51c84544f999f2454750972d34(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:06,869 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:06,869 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=13, startTime=1680321486384; duration=0sec 2023-04-01 03:58:06,869 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:06,876 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/08c7f24386e240f5b0440ea15e35b31b 2023-04-01 03:58:06,916 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/2c73568b711f44029aa053ce8bb180c5 2023-04-01 03:58:06,976 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/212eaeaab8d1424fb82a7559c75d356d 2023-04-01 03:58:06,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/08c7f24386e240f5b0440ea15e35b31b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b 2023-04-01 03:58:06,987 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b, entries=150, sequenceid=206, filesize=25.3 K 2023-04-01 03:58:06,988 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/2c73568b711f44029aa053ce8bb180c5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5 2023-04-01 03:58:06,991 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5, entries=150, sequenceid=206, filesize=11.8 K 2023-04-01 03:58:06,992 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/212eaeaab8d1424fb82a7559c75d356d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d 2023-04-01 03:58:06,995 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d, entries=150, sequenceid=206, filesize=11.8 K 2023-04-01 03:58:06,996 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c34029273b14edd6bcd250b04ada3f0e in 558ms, sequenceid=206, compaction requested=false 2023-04-01 03:58:06,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:06,996 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:06,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:06,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:06,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:06,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:06,997 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:06,998 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:06,998 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:06,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:06,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:06,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:06,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:06,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:06,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21bee659[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:06,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:06,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,006 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,006 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,007 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,007 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,010 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,012 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,012 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:07,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:07,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:07,068 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040108c5d71058ea4d1fb56eb92859610e14 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040108c5d71058ea4d1fb56eb92859610e14 2023-04-01 03:58:07,075 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/a44978ee773a4d6697874005221962a7 2023-04-01 03:58:07,112 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,117 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,122 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,125 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,125 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,125 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,125 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf32e88[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,128 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,128 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,130 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,130 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,132 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,135 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,135 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,135 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,135 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,135 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,137 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,137 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,137 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321547131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,138 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,146 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,147 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321547147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321547147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321547147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321547148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321547239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,247 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,254 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,254 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,261 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,261 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,261 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,268 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,284 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aed25f8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,285 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,285 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,286 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,286 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,292 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,295 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,295 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,300 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,300 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,301 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,305 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,305 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,405 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,414 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,414 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,414 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,414 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,417 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,417 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,425 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,425 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7246298b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,431 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,437 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,437 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,437 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,437 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,437 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,445 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,445 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,445 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,446 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,446 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321547454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/a5f06a5c5f6645038f5e139772914496 2023-04-01 03:58:07,546 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,556 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,557 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,557 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,557 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,558 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249c85b4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,563 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,563 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,563 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,563 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,564 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,564 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,564 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,564 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,564 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,566 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,570 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,570 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,570 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321547658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321547659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321547661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321547662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,672 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,678 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,679 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,686 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,686 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,686 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,686 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,688 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,688 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615e68c1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,691 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,693 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,693 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,694 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,694 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,694 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,695 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,698 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,698 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,698 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,698 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:07,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321547761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,799 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,801 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,802 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,802 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,803 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,803 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,806 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,806 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,806 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,806 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff43f7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,811 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,811 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,811 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,813 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:07,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,814 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,814 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,817 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,818 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,820 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,820 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,820 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,820 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,920 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:07,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:07,938 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:07,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/34763fba56614a9dad8e498d0cb06514 2023-04-01 03:58:07,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:07,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:07,939 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,939 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,939 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:07,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,941 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:07,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:07,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:07,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:07,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:07,941 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:07,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/a44978ee773a4d6697874005221962a7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7 2023-04-01 03:58:07,947 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:07,947 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:07,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:07,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:07,957 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:07,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:07,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:07,957 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:07,960 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,960 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:07,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b63852[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:07,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:07,962 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,963 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:07,965 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:07,965 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:07,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:07,966 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:07,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:07,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:07,966 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:07,966 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:07,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:07,967 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,968 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7, entries=200, sequenceid=224, filesize=32.1 K 2023-04-01 03:58:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/a5f06a5c5f6645038f5e139772914496 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496 2023-04-01 03:58:07,971 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:07,973 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:07,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:07,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:07,973 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:07,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:07,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:07,974 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,974 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:07,974 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:07,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:07,974 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:07,974 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:07,974 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:07,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:07,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496, entries=150, sequenceid=224, filesize=11.8 K 2023-04-01 03:58:07,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/34763fba56614a9dad8e498d0cb06514 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514 2023-04-01 03:58:07,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514, entries=150, sequenceid=224, filesize=11.8 K 2023-04-01 03:58:07,984 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c34029273b14edd6bcd250b04ada3f0e in 951ms, sequenceid=224, compaction requested=true 2023-04-01 03:58:07,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:07,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:07,984 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:07,984 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:07,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:07,984 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:07,985 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:07,985 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:07,985 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,985 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/0bdd5c51c84544f999f2454750972d34, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=83.1 K 2023-04-01 03:58:07,986 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:07,986 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:07,986 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:07,986 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0caa94c4050d404485ce8e1f44cd94fd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.7 K 2023-04-01 03:58:07,986 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0bdd5c51c84544f999f2454750972d34, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:07,987 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0caa94c4050d404485ce8e1f44cd94fd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:07,987 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 08c7f24386e240f5b0440ea15e35b31b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680321485882 2023-04-01 03:58:07,987 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2c73568b711f44029aa053ce8bb180c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680321485882 2023-04-01 03:58:07,987 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting a44978ee773a4d6697874005221962a7, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486520 2023-04-01 03:58:07,987 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a5f06a5c5f6645038f5e139772914496, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486525 2023-04-01 03:58:08,011 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:08,015 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#336 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:08,074 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:08,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:08,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:08,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:08,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:08,078 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:08,078 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:08,078 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:08,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:08,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:08,079 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:08,083 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:08,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:08,083 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:08,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:08,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:08,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:08,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:08,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:08,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:08,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:08,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:08,092 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:08,092 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:08,093 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:08,093 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:08,135 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304017eafd24583d54fd5bd69fa02593385d0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017eafd24583d54fd5bd69fa02593385d0 2023-04-01 03:58:08,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:08,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:08,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321548337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321548442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,448 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/1e1833cbce7c4e2cb26ab191cdc8dacc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1e1833cbce7c4e2cb26ab191cdc8dacc 2023-04-01 03:58:08,455 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into 1e1833cbce7c4e2cb26ab191cdc8dacc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:08,455 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:08,455 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=13, startTime=1680321487984; duration=0sec 2023-04-01 03:58:08,455 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:08,455 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:08,456 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:08,456 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:08,456 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:08,456 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/746c04329c4a4ab2b07646999bc6e244, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.7 K 2023-04-01 03:58:08,458 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 746c04329c4a4ab2b07646999bc6e244, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1680321484686 2023-04-01 03:58:08,458 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 212eaeaab8d1424fb82a7559c75d356d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680321485882 2023-04-01 03:58:08,458 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 34763fba56614a9dad8e498d0cb06514, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486525 2023-04-01 03:58:08,463 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:08,550 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/3b81867098554ee79d0d2e07db502baa 2023-04-01 03:58:08,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321548649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41976 deadline: 1680321548664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41924 deadline: 1680321548665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321548677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321548678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,868 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/63c76930c3cc4647ae4e1441b9fa53c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/63c76930c3cc4647ae4e1441b9fa53c0 2023-04-01 03:58:08,871 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/36b766936b1c4a42883f30669199de16 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b766936b1c4a42883f30669199de16 2023-04-01 03:58:08,874 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into 36b766936b1c4a42883f30669199de16(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:08,875 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:08,875 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=13, startTime=1680321487984; duration=0sec 2023-04-01 03:58:08,875 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:08,879 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into 63c76930c3cc4647ae4e1441b9fa53c0(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:08,880 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:08,880 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=13, startTime=1680321487984; duration=0sec 2023-04-01 03:58:08,880 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:08,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:08,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321548956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:08,962 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/4143d475515447a4ac06e815cce0aefe 2023-04-01 03:58:09,377 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/48fb37c14a0046489fb025c1fae5c6f7 2023-04-01 03:58:09,380 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/3b81867098554ee79d0d2e07db502baa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa 2023-04-01 03:58:09,383 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa, entries=150, sequenceid=241, filesize=25.3 K 2023-04-01 03:58:09,384 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/4143d475515447a4ac06e815cce0aefe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe 2023-04-01 03:58:09,388 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe, entries=150, sequenceid=241, filesize=11.8 K 2023-04-01 03:58:09,388 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/48fb37c14a0046489fb025c1fae5c6f7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7 2023-04-01 03:58:09,391 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7, entries=150, sequenceid=241, filesize=11.8 K 2023-04-01 03:58:09,392 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c34029273b14edd6bcd250b04ada3f0e in 1300ms, sequenceid=241, compaction requested=false 2023-04-01 03:58:09,392 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:09,392 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:09,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:09,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:09,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7a4f5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:09,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,398 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,399 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:09,399 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:09,399 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:09,399 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,399 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:09,400 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:09,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:09,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:09,402 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:09,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:09,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:09,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:09,402 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:09,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,410 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,411 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,416 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:09,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:09,416 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:09,416 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:09,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:09,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:09,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:09,516 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:09,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:09,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:09,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:09,523 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:09,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:09,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:09,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,523 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,524 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:09,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:09,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:09,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:09,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:09,528 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,528 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:09,528 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:09,529 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:09,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:09,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:09,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:09,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:09,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134dcc6c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:09,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:09,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,544 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:09,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:09,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:09,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:09,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:09,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:09,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:09,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:09,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:09,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:09,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:09,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:09,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:09,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:09,553 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:09,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,553 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:09,554 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:09,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:09,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321549564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,654 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:09,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:09,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:09,656 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:09,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:09,656 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:09,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,661 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:09,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:09,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:09,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:09,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:09,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:09,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:09,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:09,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:09,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:09,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:09,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:09,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:09,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:09,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321549669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,690 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:09,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b55d509[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,692 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:09,692 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:09,692 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:09,693 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:09,693 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,693 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:09,694 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:09,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:09,698 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:09,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:09,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:09,699 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:09,699 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:09,717 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,717 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,717 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,717 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,725 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:09,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:09,726 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:09,726 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,726 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,726 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:09,826 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:09,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:09,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:09,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:09,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:09,828 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:09,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:09,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:09,829 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,829 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,835 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:09,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:09,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:09,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:09,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:09,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:09,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:09,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:09,846 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:09,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:09,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:09,851 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:09,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd11499[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:09,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:09,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,859 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,860 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:09,860 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:09,860 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:09,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,861 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:09,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:09,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:09,869 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:09,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:09,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:09,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:09,869 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:09,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,876 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:09,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:09,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321549876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,883 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ac26be857ff942a49b865adf3389a1a8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ac26be857ff942a49b865adf3389a1a8 2023-04-01 03:58:09,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:09,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:09,891 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:09,891 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:09,891 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:09,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:09,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:09,919 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/b2f37d331acc492cbbb98b90ea882398 2023-04-01 03:58:09,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/e65aafc5dd8449908b843e0244b4807f 2023-04-01 03:58:09,991 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:09,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:09,995 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:09,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:09,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:09,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:09,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:09,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:09,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:09,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:09,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:09,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:09,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:09,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:10,005 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:10,005 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:10,005 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,005 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:10,006 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:10,006 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,006 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:10,006 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:10,006 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,006 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:10,006 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:10,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,008 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,008 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,008 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:10,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278dfb1a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:10,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:10,010 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,011 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,011 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:10,011 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:10,011 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:10,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:10,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:10,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:10,015 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:10,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:10,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:10,015 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:10,015 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:10,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,016 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,017 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,021 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,021 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,021 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:10,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:10,021 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,022 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:10,022 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:10,122 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:10,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:10,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:10,125 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:10,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:10,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:10,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,139 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,139 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:10,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:10,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:10,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:10,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:10,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:10,159 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,159 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:10,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:10,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:10,160 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:10,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:10,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:10,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:10,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,161 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ade6d2b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:10,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:10,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:10,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:10,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:10,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:10,176 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,176 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:10,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:10,178 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:10,179 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:10,179 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:10,179 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:10,179 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:10,180 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,180 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,180 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,181 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,183 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321550184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:10,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,198 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,198 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:10,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,299 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:10,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:10,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:10,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:10,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:10,301 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:10,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:10,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:10,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:10,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:10,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:10,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:10,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:10,305 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:10,305 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:10,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:10,308 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:10,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6616e4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:10,308 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,309 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:10,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:10,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:10,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,309 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:10,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:10,313 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,313 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:10,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:10,315 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:10,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:10,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:10,315 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:10,315 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:10,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:10,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:10,318 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:10,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:10,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,320 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:10,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:10,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:10,321 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:10,321 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:10,321 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:10,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:10,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:10,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/7373823b5bba4199b70a9c896088b19c 2023-04-01 03:58:10,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/b2f37d331acc492cbbb98b90ea882398 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398 2023-04-01 03:58:10,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398, entries=200, sequenceid=260, filesize=32.2 K 2023-04-01 03:58:10,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/e65aafc5dd8449908b843e0244b4807f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f 2023-04-01 03:58:10,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f, entries=150, sequenceid=260, filesize=11.8 K 2023-04-01 03:58:10,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/7373823b5bba4199b70a9c896088b19c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c 2023-04-01 03:58:10,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c, entries=150, sequenceid=260, filesize=11.8 K 2023-04-01 03:58:10,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c34029273b14edd6bcd250b04ada3f0e in 921ms, sequenceid=260, compaction requested=true 2023-04-01 03:58:10,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:10,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:10,386 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:10,386 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:10,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:10,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:10,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:10,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:10,387 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,387 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/63c76930c3cc4647ae4e1441b9fa53c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=83.3 K 2023-04-01 03:58:10,387 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 63c76930c3cc4647ae4e1441b9fa53c0, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486525 2023-04-01 03:58:10,387 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:10,387 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:10,387 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,388 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1e1833cbce7c4e2cb26ab191cdc8dacc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.8 K 2023-04-01 03:58:10,389 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 3b81867098554ee79d0d2e07db502baa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321487103 2023-04-01 03:58:10,389 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b2f37d331acc492cbbb98b90ea882398, keycount=200, bloomtype=ROW, size=32.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488313 2023-04-01 03:58:10,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1e1833cbce7c4e2cb26ab191cdc8dacc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486525 2023-04-01 03:58:10,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4143d475515447a4ac06e815cce0aefe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321487103 2023-04-01 03:58:10,391 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e65aafc5dd8449908b843e0244b4807f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488320 2023-04-01 03:58:10,411 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:10,421 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:10,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:10,426 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:10,426 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,426 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:10,426 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:10,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:10,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,428 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:10,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:10,431 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#346 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:10,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:10,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:10,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:10,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:10,489 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,489 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:10,489 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:58:10,489 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:10,490 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:10,490 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:10,490 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:10,490 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:10,490 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:10,550 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401d6b5d251c9f3413f8861681703fb7a81 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d6b5d251c9f3413f8861681703fb7a81 2023-04-01 03:58:10,582 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/f5a047a029314f60a5b197db68a188b6 2023-04-01 03:58:10,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:10,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:10,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321550743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321550757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321550759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321550761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41980 deadline: 1680321550762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,840 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/d56dcba0145a47b4886e352d4638361f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/d56dcba0145a47b4886e352d4638361f 2023-04-01 03:58:10,846 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into d56dcba0145a47b4886e352d4638361f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:10,847 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:10,847 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=13, startTime=1680321490386; duration=0sec 2023-04-01 03:58:10,847 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:10,847 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:10,848 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:10,848 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:10,848 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:10,848 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b766936b1c4a42883f30669199de16, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=35.8 K 2023-04-01 03:58:10,848 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 36b766936b1c4a42883f30669199de16, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680321486525 2023-04-01 03:58:10,849 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 48fb37c14a0046489fb025c1fae5c6f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321487103 2023-04-01 03:58:10,849 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7373823b5bba4199b70a9c896088b19c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488320 2023-04-01 03:58:10,854 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#349 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:10,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41908 deadline: 1680321550863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321550874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321550875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321550875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:10,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321550886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:10,900 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/a1aae5e9e15f465b8e1545665a58181d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/a1aae5e9e15f465b8e1545665a58181d 2023-04-01 03:58:10,911 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into a1aae5e9e15f465b8e1545665a58181d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:10,911 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:10,911 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=13, startTime=1680321490386; duration=0sec 2023-04-01 03:58:10,911 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:10,925 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/b429e490f8e44ab7808c617e7aa6bc4b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b429e490f8e44ab7808c617e7aa6bc4b 2023-04-01 03:58:10,930 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into b429e490f8e44ab7808c617e7aa6bc4b(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:10,930 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:10,930 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=13, startTime=1680321490386; duration=0sec 2023-04-01 03:58:10,931 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:11,012 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/cd6177dd1d7b4cebbfa8777ff5743bf2 2023-04-01 03:58:11,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321551073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321551081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321551081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321551084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321551094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321551380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321551388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321551389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321551390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321551400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,493 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/e9e507b6a9464c1097702ef78f3d9085 2023-04-01 03:58:11,496 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/f5a047a029314f60a5b197db68a188b6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6 2023-04-01 03:58:11,499 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6, entries=100, sequenceid=276, filesize=18.6 K 2023-04-01 03:58:11,499 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/cd6177dd1d7b4cebbfa8777ff5743bf2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2 2023-04-01 03:58:11,503 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2, entries=100, sequenceid=276, filesize=9.5 K 2023-04-01 03:58:11,503 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/e9e507b6a9464c1097702ef78f3d9085 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085 2023-04-01 03:58:11,507 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085, entries=100, sequenceid=276, filesize=9.5 K 2023-04-01 03:58:11,507 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for c34029273b14edd6bcd250b04ada3f0e in 1018ms, sequenceid=276, compaction requested=false 2023-04-01 03:58:11,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:11,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:11,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:11,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:11,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:11,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:11,510 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,510 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:11,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:11,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:11,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c80ac4d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:11,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:11,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:11,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:11,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:11,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:11,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:11,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:11,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:11,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:11,544 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:11,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:11,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:11,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:11,544 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:11,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:11,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:11,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:11,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:11,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:11,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:11,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:11,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:11,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:11,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:11,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,559 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:11,559 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:11,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:11,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:11,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:11,659 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:11,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:11,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:11,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:11,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:11,662 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:11,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,663 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:11,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:11,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:11,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:11,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:11,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:11,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:11,666 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:11,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:11,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:11,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:11,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:11,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:11,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321551906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321551907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321551909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321551911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:11,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:11,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321551912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321552013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321552013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321552014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321552021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321552022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,082 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401aa0ab99fa08941fbb73a6392f11217d1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aa0ab99fa08941fbb73a6392f11217d1 2023-04-01 03:58:12,086 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/a83d1d4a672f4f029135ef48ee7d4c17 2023-04-01 03:58:12,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321552216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321552216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321552220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321552226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321552226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,495 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/91a60177ef3146009452dd3be644aa64 2023-04-01 03:58:12,517 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/863ecd13e6db46f39d8f37899914d2a5 2023-04-01 03:58:12,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/a83d1d4a672f4f029135ef48ee7d4c17 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17 2023-04-01 03:58:12,525 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17, entries=150, sequenceid=296, filesize=25.4 K 2023-04-01 03:58:12,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/91a60177ef3146009452dd3be644aa64 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64 2023-04-01 03:58:12,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321552525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321552525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,531 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64, entries=150, sequenceid=296, filesize=11.9 K 2023-04-01 03:58:12,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321552526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,532 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/863ecd13e6db46f39d8f37899914d2a5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5 2023-04-01 03:58:12,535 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5, entries=150, sequenceid=296, filesize=11.9 K 2023-04-01 03:58:12,535 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c34029273b14edd6bcd250b04ada3f0e in 869ms, sequenceid=296, compaction requested=true 2023-04-01 03:58:12,535 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:12,535 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:12,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:12,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:12,535 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:12,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:12,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:58:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:12,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:12,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@441dd3ec[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:12,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:12,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:12,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:12,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:12,545 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:12,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:12,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:12,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:12,545 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:12,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,547 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,547 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:12,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:12,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:12,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,551 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:12,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,551 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:12,551 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040181e2359846ab46a99a2de3af4123edc3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181e2359846ab46a99a2de3af4123edc3 2023-04-01 03:58:12,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321552577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321552577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,651 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:12,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:12,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:12,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:12,653 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:12,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:12,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:12,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:12,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:12,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:12,656 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,656 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:12,656 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:12,656 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:12,656 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:12,658 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:12,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:12,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d834971[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:12,659 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,661 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:12,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:12,662 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:12,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:12,664 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:12,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:12,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:12,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:12,664 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:12,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,667 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,670 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,670 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:12,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,671 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,671 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:12,671 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:12,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321552680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321552680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,771 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:12,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:12,774 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:12,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:12,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:12,775 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,775 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,776 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,776 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,776 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:12,776 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:12,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:12,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:12,778 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:12,778 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,778 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:12,786 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,786 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:12,786 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:12,786 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:12,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:12,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:12,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42fdb313[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:12,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:12,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:12,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:12,800 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:12,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:12,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:12,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:12,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:12,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:12,804 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:12,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:12,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:12,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:12,804 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:12,806 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,806 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,807 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,808 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:12,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:12,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,810 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:12,810 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321552887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:12,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321552887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,910 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:12,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:12,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:12,913 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:12,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:12,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:12,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,920 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:12,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:12,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:12,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:12,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:12,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:12,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:12,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,942 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:12,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:12,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b169dc5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:12,943 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:12,944 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,944 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:12,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:12,944 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:12,945 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:12,945 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,945 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,945 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:12,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:12,947 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:12,947 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:12,947 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:12,947 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:12,947 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:12,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:12,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,948 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,948 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:12,949 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:12,950 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:12,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:12,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:12,950 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:12,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:12,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:12,951 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:12,951 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:12,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:12,954 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=313, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/10df032504c2434ba8d9f577b2c89922 2023-04-01 03:58:13,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321553033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321553033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321553034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,051 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,053 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,056 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,057 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,057 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61bea5f8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,059 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,059 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,059 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,065 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,066 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,066 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,066 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,073 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,073 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,073 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,173 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,178 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,179 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,179 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,184 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,184 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,186 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,186 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,194 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,195 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1681a07f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321553194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321553194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,201 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,201 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,202 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,202 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,202 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,202 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,204 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,204 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,204 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,206 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,207 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,209 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,210 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,211 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,211 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,211 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,311 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,314 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,314 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,316 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,320 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,322 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394bfc97[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,322 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,323 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,323 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,323 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,323 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,323 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,323 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,326 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,327 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,328 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,330 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,332 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,335 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,335 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,335 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/761907cd5b6e4beeb41b9ac813b18564 2023-04-01 03:58:13,436 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,438 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,441 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,442 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,442 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,442 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,442 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464cb6ce[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,450 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,450 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,451 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,451 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,451 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,452 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,454 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,454 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,555 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,558 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,572 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,572 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,572 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,572 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,583 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,583 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,584 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,584 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52495a9a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,585 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,585 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,586 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,586 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,586 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,586 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,588 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,588 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,588 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,617 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,617 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,619 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,619 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,619 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321553699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:13,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321553702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,720 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,724 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,725 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,726 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,730 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,730 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,730 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:13,730 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:13,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:13,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49508af4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:13,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:13,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,738 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:13,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:13,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:13,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,742 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:13,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:13,746 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:13,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:13,747 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:13,747 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:13,747 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:13,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:13,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:13,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,752 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:13,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:13,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:13,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:13,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:13,761 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:13,761 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:13,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:13,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:13,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/c5430226d9494d6e93b8d1518afdcdfd 2023-04-01 03:58:13,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/10df032504c2434ba8d9f577b2c89922 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922 2023-04-01 03:58:13,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922, entries=250, sequenceid=313, filesize=39.2 K 2023-04-01 03:58:13,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/761907cd5b6e4beeb41b9ac813b18564 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564 2023-04-01 03:58:13,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564, entries=150, sequenceid=313, filesize=11.9 K 2023-04-01 03:58:13,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/c5430226d9494d6e93b8d1518afdcdfd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd 2023-04-01 03:58:13,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd, entries=150, sequenceid=313, filesize=11.9 K 2023-04-01 03:58:13,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c34029273b14edd6bcd250b04ada3f0e in 1261ms, sequenceid=313, compaction requested=true 2023-04-01 03:58:13,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:13,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:13,799 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:13,799 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:13,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:13,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:13,801 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:13,801 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:13,801 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,801 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/d56dcba0145a47b4886e352d4638361f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=45.8 K 2023-04-01 03:58:13,802 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111768 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:13,803 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:13,803 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,803 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b429e490f8e44ab7808c617e7aa6bc4b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=109.1 K 2023-04-01 03:58:13,803 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d56dcba0145a47b4886e352d4638361f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488320 2023-04-01 03:58:13,803 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b429e490f8e44ab7808c617e7aa6bc4b, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488320 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f5a047a029314f60a5b197db68a188b6, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321489547 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting cd6177dd1d7b4cebbfa8777ff5743bf2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321489547 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting a83d1d4a672f4f029135ef48ee7d4c17, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680321490756 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 91a60177ef3146009452dd3be644aa64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680321490756 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 10df032504c2434ba8d9f577b2c89922, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:13,804 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 761907cd5b6e4beeb41b9ac813b18564, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:13,815 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:13,840 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#358 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:13,861 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:13,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:13,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:13,866 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/b6207c8fd4e7446a976b95ad4f730e1f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b6207c8fd4e7446a976b95ad4f730e1f 2023-04-01 03:58:13,876 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into b6207c8fd4e7446a976b95ad4f730e1f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:13,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:13,876 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=12, startTime=1680321493799; duration=0sec 2023-04-01 03:58:13,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:13,876 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:13,877 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:13,877 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:13,877 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,877 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/a1aae5e9e15f465b8e1545665a58181d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=45.8 K 2023-04-01 03:58:13,878 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a1aae5e9e15f465b8e1545665a58181d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680321488320 2023-04-01 03:58:13,878 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e9e507b6a9464c1097702ef78f3d9085, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680321489547 2023-04-01 03:58:13,878 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 863ecd13e6db46f39d8f37899914d2a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680321490756 2023-04-01 03:58:13,878 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c5430226d9494d6e93b8d1518afdcdfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:13,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:13,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:13,882 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:13,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:13,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:13,883 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,883 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:13,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:13,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:13,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:13,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:13,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:13,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:13,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:13,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:13,886 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:13,888 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:13,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:13,895 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:13,897 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:13,897 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:13,897 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:58:13,897 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:13,897 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:13,898 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:13,898 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:13,898 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:13,898 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:13,916 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/11f02e83bc2b4ae4b344fecda7adc069 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/11f02e83bc2b4ae4b344fecda7adc069 2023-04-01 03:58:13,923 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into 11f02e83bc2b4ae4b344fecda7adc069(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:13,923 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:13,923 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=12, startTime=1680321493800; duration=0sec 2023-04-01 03:58:13,923 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:14,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:14,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321554068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321554074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321554074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321554175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321554179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321554179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,339 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018384750151e64f899fcc5dcf283f9472 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018384750151e64f899fcc5dcf283f9472 2023-04-01 03:58:14,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321554380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321554385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321554385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321554687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321554692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321554694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321554712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321554715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:14,743 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c5fc68d6bd6a49d7b6b84f362282daa0 2023-04-01 03:58:15,084 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/f5a9e739e122434a944840f89632962d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a9e739e122434a944840f89632962d 2023-04-01 03:58:15,087 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into f5a9e739e122434a944840f89632962d(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:58:15,087 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:15,087 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=12, startTime=1680321493799; duration=1sec 2023-04-01 03:58:15,087 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:15,152 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/6502ce9d9e4a455c881178f94c820d6d 2023-04-01 03:58:15,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:15,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321555192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:15,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321555196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:15,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321555200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,563 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5be1675010d54439bb51729975ecb19c 2023-04-01 03:58:15,567 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/c5fc68d6bd6a49d7b6b84f362282daa0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0 2023-04-01 03:58:15,569 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0, entries=150, sequenceid=329, filesize=25.4 K 2023-04-01 03:58:15,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/6502ce9d9e4a455c881178f94c820d6d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d 2023-04-01 03:58:15,573 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d, entries=150, sequenceid=329, filesize=11.9 K 2023-04-01 03:58:15,574 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5be1675010d54439bb51729975ecb19c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c 2023-04-01 03:58:15,586 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c, entries=150, sequenceid=329, filesize=11.9 K 2023-04-01 03:58:15,587 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for c34029273b14edd6bcd250b04ada3f0e in 1690ms, sequenceid=329, compaction requested=false 2023-04-01 03:58:15,587 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:15,587 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:15,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:15,589 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:15,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:15,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f99780d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:15,590 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:15,590 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:15,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:15,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:15,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:15,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:15,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:15,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:15,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:15,595 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:15,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:15,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:15,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:15,595 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:15,596 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,596 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:15,596 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:15,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:15,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:15,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:15,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:15,600 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:15,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:15,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:15,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:15,601 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:15,601 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:15,701 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:15,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:15,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:15,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:15,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:15,704 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:15,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:15,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:15,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:15,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,704 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:15,706 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:15,706 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:15,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:15,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:15,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:15,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:15,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:15,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:15,710 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:15,710 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:15,710 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:15,718 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:15,718 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. started... 2023-04-01 03:58:15,718 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:58:15,718 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:15,718 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:15,719 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:15,719 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:15,719 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:15,719 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:16,133 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011c2bf0d9bb1246f582dc131b12d95cbe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011c2bf0d9bb1246f582dc131b12d95cbe 2023-04-01 03:58:16,137 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/38da0b56c6d14e4b8d2dad210e5e6fc6 2023-04-01 03:58:16,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:16,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. as already flushing 2023-04-01 03:58:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321556230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321556230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321556232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,269 DEBUG [Thread-2189] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x651d3d36 to 127.0.0.1:50457 2023-04-01 03:58:16,269 DEBUG [Thread-2189] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:16,273 DEBUG [Thread-2193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f0877a9 to 127.0.0.1:50457 2023-04-01 03:58:16,273 DEBUG [Thread-2193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:16,275 DEBUG [Thread-2197] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06e48d7b to 127.0.0.1:50457 2023-04-01 03:58:16,275 DEBUG [Thread-2197] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:16,275 DEBUG [Thread-2195] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e0b4412 to 127.0.0.1:50457 2023-04-01 03:58:16,275 DEBUG [Thread-2195] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:16,279 DEBUG [Thread-2191] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bd89b96 to 127.0.0.1:50457 2023-04-01 03:58:16,279 DEBUG [Thread-2191] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321556339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321556339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321556339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321556543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321556542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321556543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,546 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/0efca418804d4bccbd27ddc956ea5941 2023-04-01 03:58:16,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41926 deadline: 1680321556718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,720 DEBUG [Thread-2182] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:16,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41976 deadline: 1680321556722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,724 DEBUG [Thread-2184] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:16,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41908 deadline: 1680321556846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41924 deadline: 1680321556848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41980 deadline: 1680321556848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,964 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/fb3e5465a6d2434194bd3ea67c4ae196 2023-04-01 03:58:16,972 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/38da0b56c6d14e4b8d2dad210e5e6fc6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6 2023-04-01 03:58:16,978 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6, entries=150, sequenceid=349, filesize=25.4 K 2023-04-01 03:58:16,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/0efca418804d4bccbd27ddc956ea5941 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941 2023-04-01 03:58:16,984 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941, entries=150, sequenceid=349, filesize=11.9 K 2023-04-01 03:58:16,985 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/fb3e5465a6d2434194bd3ea67c4ae196 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196 2023-04-01 03:58:16,991 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196, entries=150, sequenceid=349, filesize=11.9 K 2023-04-01 03:58:16,991 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c34029273b14edd6bcd250b04ada3f0e in 1273ms, sequenceid=349, compaction requested=true 2023-04-01 03:58:16,992 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:16,992 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:16,994 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:16,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,994 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:16,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:16,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc051f6[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:16,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:16,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,999 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:16,999 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:16,999 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:16,999 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:16,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:16,999 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:17,000 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:17,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:17,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:17,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:17,005 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:17,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:17,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:17,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:17,005 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:17,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:17,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:17,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:17,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,011 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,011 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:17,011 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:17,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:17,011 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:17,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:17,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:17,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:17,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:17,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:17,012 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:17,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:17,012 DEBUG [Thread-2188] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:17,012 DEBUG [Thread-2188] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:17,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:17,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:17,112 DEBUG [Thread-2188] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:17,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:17,349 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-01 03:58:17,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:17,368 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:58:17,369 DEBUG [Thread-2178] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x566231bc to 127.0.0.1:50457 2023-04-01 03:58:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:17,370 DEBUG [Thread-2186] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50e948f8 to 127.0.0.1:50457 2023-04-01 03:58:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:17,370 DEBUG [Thread-2180] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52f2f9ef to 127.0.0.1:50457 2023-04-01 03:58:17,370 DEBUG [Thread-2178] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:17,371 DEBUG [Thread-2180] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:17,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:17,370 DEBUG [Thread-2186] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:17,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:17,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:17,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:17,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/2c64965c8cb04826a3d8d1c96284be0d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/50b16efdbfc74db6bff9d85aa73ff809, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/0bdd5c51c84544f999f2454750972d34, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/63c76930c3cc4647ae4e1441b9fa53c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b429e490f8e44ab7808c617e7aa6bc4b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922] to archive 2023-04-01 03:58:17,391 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:17,392 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7aad55ec3c59422892bac628e01bdd97 2023-04-01 03:58:17,393 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/25847fe9d1a64dd6a965bff0c4f902e8 2023-04-01 03:58:17,396 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/668378d789d54d7f9cca5f9d6a3ea0f1 2023-04-01 03:58:17,397 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d29209908f3b4465807b2c7e569c0286 2023-04-01 03:58:17,398 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c7394271dd7240ecb97598391cea51fa 2023-04-01 03:58:17,399 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/2c64965c8cb04826a3d8d1c96284be0d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/2c64965c8cb04826a3d8d1c96284be0d 2023-04-01 03:58:17,400 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/30413ca1ed194239bc758e6afdb9d64c 2023-04-01 03:58:17,401 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c54c992df0ba49a1bf834c0de6627f51 2023-04-01 03:58:17,402 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/9429f047e0ef4ff7be0f9b62af4544d3 2023-04-01 03:58:17,403 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ac8002d897ca42daaa37307f30536d59 2023-04-01 03:58:17,404 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/50b16efdbfc74db6bff9d85aa73ff809 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/50b16efdbfc74db6bff9d85aa73ff809 2023-04-01 03:58:17,406 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f730155171f842f9a4d281a7dab15617 2023-04-01 03:58:17,407 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/ab166906ed1849d1994ab02819af6b18 2023-04-01 03:58:17,408 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/0bdd5c51c84544f999f2454750972d34 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/0bdd5c51c84544f999f2454750972d34 2023-04-01 03:58:17,409 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/08c7f24386e240f5b0440ea15e35b31b 2023-04-01 03:58:17,410 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401a9969d7118b04b399e30ca9eeab4fef5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a9969d7118b04b399e30ca9eeab4fef5 2023-04-01 03:58:17,411 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a44978ee773a4d6697874005221962a7 2023-04-01 03:58:17,412 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/63c76930c3cc4647ae4e1441b9fa53c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/63c76930c3cc4647ae4e1441b9fa53c0 2023-04-01 03:58:17,413 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/3b81867098554ee79d0d2e07db502baa 2023-04-01 03:58:17,414 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b2f37d331acc492cbbb98b90ea882398 2023-04-01 03:58:17,415 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b429e490f8e44ab7808c617e7aa6bc4b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/b429e490f8e44ab7808c617e7aa6bc4b 2023-04-01 03:58:17,416 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a047a029314f60a5b197db68a188b6 2023-04-01 03:58:17,416 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/a83d1d4a672f4f029135ef48ee7d4c17 2023-04-01 03:58:17,417 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/10df032504c2434ba8d9f577b2c89922 2023-04-01 03:58:17,421 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b38e69387d1d4129ad0fd861f59f442d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/391cc9cd27364a39af5e28fb0523f87b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0caa94c4050d404485ce8e1f44cd94fd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1e1833cbce7c4e2cb26ab191cdc8dacc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/d56dcba0145a47b4886e352d4638361f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564] to archive 2023-04-01 03:58:17,422 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:17,423 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/fbd34abc2a5745488b3507438de19011 2023-04-01 03:58:17,424 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/34b036c0309d434d95319036bc7b9470 2023-04-01 03:58:17,425 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/282fb8b9f4134f68969833e7d59f9599 2023-04-01 03:58:17,426 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/f00c1d9f32f24498a3b95984497141db 2023-04-01 03:58:17,427 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/7efcc0541003476982f80e5a02f67e71 2023-04-01 03:58:17,428 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b38e69387d1d4129ad0fd861f59f442d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b38e69387d1d4129ad0fd861f59f442d 2023-04-01 03:58:17,429 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/ac246290fd9b46e2aa75c14107fb905b 2023-04-01 03:58:17,430 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c751a53063e9417fb8a1970386e86082 2023-04-01 03:58:17,430 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0207b00376844b9292440ae0d93febb4 2023-04-01 03:58:17,432 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/391cc9cd27364a39af5e28fb0523f87b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/391cc9cd27364a39af5e28fb0523f87b 2023-04-01 03:58:17,433 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/65db88baf7b34e64b9be621bc4e233dc 2023-04-01 03:58:17,434 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1f0fe36a98734b8998babce102f63658 2023-04-01 03:58:17,435 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0caa94c4050d404485ce8e1f44cd94fd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0caa94c4050d404485ce8e1f44cd94fd 2023-04-01 03:58:17,436 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/c156402bf9614d1b984946bcb6f69433 2023-04-01 03:58:17,436 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/2c73568b711f44029aa053ce8bb180c5 2023-04-01 03:58:17,438 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1e1833cbce7c4e2cb26ab191cdc8dacc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/1e1833cbce7c4e2cb26ab191cdc8dacc 2023-04-01 03:58:17,439 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/a5f06a5c5f6645038f5e139772914496 2023-04-01 03:58:17,440 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4143d475515447a4ac06e815cce0aefe 2023-04-01 03:58:17,440 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/d56dcba0145a47b4886e352d4638361f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/d56dcba0145a47b4886e352d4638361f 2023-04-01 03:58:17,441 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e65aafc5dd8449908b843e0244b4807f 2023-04-01 03:58:17,442 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/cd6177dd1d7b4cebbfa8777ff5743bf2 2023-04-01 03:58:17,443 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/91a60177ef3146009452dd3be644aa64 2023-04-01 03:58:17,444 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/761907cd5b6e4beeb41b9ac813b18564 2023-04-01 03:58:17,448 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/6881efc053c1454c83308866122e13e3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c42568f75dc64756a75134ea8af4cedb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/746c04329c4a4ab2b07646999bc6e244, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b766936b1c4a42883f30669199de16, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/a1aae5e9e15f465b8e1545665a58181d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd] to archive 2023-04-01 03:58:17,449 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:17,450 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/b15e8f5e53404a7b9c6429f0c8ecce75 2023-04-01 03:58:17,451 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/3e1edd0c997c4837bd066c6f3dedbeb1 2023-04-01 03:58:17,452 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/445d7900182a4c77980fc9521a1c2eb8 2023-04-01 03:58:17,452 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/0be65dd9e7b1471ab56ce7bc42505f2c 2023-04-01 03:58:17,453 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b05261ef23447a86cad0fa31418e0a 2023-04-01 03:58:17,454 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/6881efc053c1454c83308866122e13e3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/6881efc053c1454c83308866122e13e3 2023-04-01 03:58:17,455 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/8394761f986241bdaeac8c146c7028d1 2023-04-01 03:58:17,456 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/66b32c3126064207a4c8dd57d40cdac3 2023-04-01 03:58:17,457 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c6f3cb33546465a8df9af232e7e2927 2023-04-01 03:58:17,457 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c42568f75dc64756a75134ea8af4cedb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c42568f75dc64756a75134ea8af4cedb 2023-04-01 03:58:17,458 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/f9e969ec4d65410b8c935ecb17daad39 2023-04-01 03:58:17,459 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/719ca80c55e14a92ba12c9926a48ef92 2023-04-01 03:58:17,460 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/746c04329c4a4ab2b07646999bc6e244 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/746c04329c4a4ab2b07646999bc6e244 2023-04-01 03:58:17,460 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e70aa956a2fc45399d9a8c11a8d52a20 2023-04-01 03:58:17,461 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/212eaeaab8d1424fb82a7559c75d356d 2023-04-01 03:58:17,461 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b766936b1c4a42883f30669199de16 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/36b766936b1c4a42883f30669199de16 2023-04-01 03:58:17,462 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/34763fba56614a9dad8e498d0cb06514 2023-04-01 03:58:17,463 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/48fb37c14a0046489fb025c1fae5c6f7 2023-04-01 03:58:17,464 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/a1aae5e9e15f465b8e1545665a58181d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/a1aae5e9e15f465b8e1545665a58181d 2023-04-01 03:58:17,464 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/7373823b5bba4199b70a9c896088b19c 2023-04-01 03:58:17,465 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/e9e507b6a9464c1097702ef78f3d9085 2023-04-01 03:58:17,466 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/863ecd13e6db46f39d8f37899914d2a5 2023-04-01 03:58:17,467 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/c5430226d9494d6e93b8d1518afdcdfd 2023-04-01 03:58:17,816 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/7cab7d6bfd3d4e64a3a371d15addefc2 2023-04-01 03:58:18,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/4f16dce0299347d2886abf2a8312e78a 2023-04-01 03:58:18,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5c4f42da3e104228bf40d2c0799caad6 2023-04-01 03:58:18,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/7cab7d6bfd3d4e64a3a371d15addefc2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2 2023-04-01 03:58:18,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2, entries=150, sequenceid=366, filesize=25.4 K 2023-04-01 03:58:18,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/4f16dce0299347d2886abf2a8312e78a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a 2023-04-01 03:58:18,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a, entries=150, sequenceid=366, filesize=11.9 K 2023-04-01 03:58:18,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5c4f42da3e104228bf40d2c0799caad6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6 2023-04-01 03:58:18,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6, entries=150, sequenceid=366, filesize=11.9 K 2023-04-01 03:58:18,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for c34029273b14edd6bcd250b04ada3f0e in 891ms, sequenceid=366, compaction requested=true 2023-04-01 03:58:18,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:18,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:18,257 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:18,257 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:18,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:18,257 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:18,258 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:18,258 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/A is initiating minor compaction (all files) 2023-04-01 03:58:18,258 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/A in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:18,259 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a9e739e122434a944840f89632962d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=102.5 K 2023-04-01 03:58:18,259 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:18,259 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f5a9e739e122434a944840f89632962d, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:18,259 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/B is initiating minor compaction (all files) 2023-04-01 03:58:18,259 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/B in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:18,259 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b6207c8fd4e7446a976b95ad4f730e1f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=48.4 K 2023-04-01 03:58:18,259 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c5fc68d6bd6a49d7b6b84f362282daa0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680321492570 2023-04-01 03:58:18,260 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b6207c8fd4e7446a976b95ad4f730e1f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:18,260 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 38da0b56c6d14e4b8d2dad210e5e6fc6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680321494066 2023-04-01 03:58:18,260 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6502ce9d9e4a455c881178f94c820d6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680321492570 2023-04-01 03:58:18,260 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7cab7d6bfd3d4e64a3a371d15addefc2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680321496227 2023-04-01 03:58:18,260 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0efca418804d4bccbd27ddc956ea5941, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680321494066 2023-04-01 03:58:18,261 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4f16dce0299347d2886abf2a8312e78a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680321496227 2023-04-01 03:58:18,274 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:18,280 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#A#compaction#369 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:18,697 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/85d88184876a4d9aa72e18f7554a1b51 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/85d88184876a4d9aa72e18f7554a1b51 2023-04-01 03:58:18,700 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/B of c34029273b14edd6bcd250b04ada3f0e into 85d88184876a4d9aa72e18f7554a1b51(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:18,700 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:18,700 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/B, priority=12, startTime=1680321498257; duration=0sec 2023-04-01 03:58:18,700 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:18,700 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:18,701 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:18,701 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): c34029273b14edd6bcd250b04ada3f0e/C is initiating minor compaction (all files) 2023-04-01 03:58:18,702 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c34029273b14edd6bcd250b04ada3f0e/C in TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:18,702 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/11f02e83bc2b4ae4b344fecda7adc069, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp, totalSize=48.4 K 2023-04-01 03:58:18,702 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 11f02e83bc2b4ae4b344fecda7adc069, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1680321491906 2023-04-01 03:58:18,702 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 5be1675010d54439bb51729975ecb19c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680321492570 2023-04-01 03:58:18,702 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fb3e5465a6d2434194bd3ea67c4ae196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680321494066 2023-04-01 03:58:18,703 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 5c4f42da3e104228bf40d2c0799caad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1680321496227 2023-04-01 03:58:18,709 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): c34029273b14edd6bcd250b04ada3f0e#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:18,710 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/54efed2278f144868bb37d316e5c52a1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/54efed2278f144868bb37d316e5c52a1 2023-04-01 03:58:18,713 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/A of c34029273b14edd6bcd250b04ada3f0e into 54efed2278f144868bb37d316e5c52a1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:18,713 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:18,713 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/A, priority=12, startTime=1680321498257; duration=0sec 2023-04-01 03:58:18,713 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:19,117 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/5cd1d93eb4d046228b39c3d7a2066f7c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5cd1d93eb4d046228b39c3d7a2066f7c 2023-04-01 03:58:19,123 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c34029273b14edd6bcd250b04ada3f0e/C of c34029273b14edd6bcd250b04ada3f0e into 5cd1d93eb4d046228b39c3d7a2066f7c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:19,123 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:19,123 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e., storeName=c34029273b14edd6bcd250b04ada3f0e/C, priority=12, startTime=1680321498257; duration=0sec 2023-04-01 03:58:19,124 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:20,744 DEBUG [Thread-2184] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58a97f1f to 127.0.0.1:50457 2023-04-01 03:58:20,744 DEBUG [Thread-2184] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:20,745 DEBUG [Thread-2182] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142c61b8 to 127.0.0.1:50457 2023-04-01 03:58:20,745 DEBUG [Thread-2182] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1820 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5452 rows 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1810 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5428 rows 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1830 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5489 rows 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1810 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5429 rows 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1828 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5479 rows 2023-04-01 03:58:20,745 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:58:20,745 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4dc83d7d to 127.0.0.1:50457 2023-04-01 03:58:20,746 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:20,747 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:58:20,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:58:20,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:20,754 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321500753"}]},"ts":"1680321500753"} 2023-04-01 03:58:20,755 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:58:20,766 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:58:20,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, UNASSIGN}] 2023-04-01 03:58:20,768 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, UNASSIGN 2023-04-01 03:58:20,769 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:20,769 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321500769"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321500769"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321500769"}]},"ts":"1680321500769"} 2023-04-01 03:58:20,770 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:20,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:20,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c34029273b14edd6bcd250b04ada3f0e, disabling compactions & flushes 2023-04-01 03:58:20,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. after waiting 0 ms 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:20,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c34029273b14edd6bcd250b04ada3f0e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=A 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=B 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c34029273b14edd6bcd250b04ada3f0e, store=C 2023-04-01 03:58:20,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:21,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:21,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040137760327e7d94d2b838861df5952d7ac to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040137760327e7d94d2b838861df5952d7ac 2023-04-01 03:58:21,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/d20217cf9a5448debcf10ce3301d1d39 2023-04-01 03:58:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:21,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/e5f52bb8463045bdb73907b3a505f3be 2023-04-01 03:58:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:22,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/cff1216dd9054c29af234616449b9b5d 2023-04-01 03:58:22,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/A/d20217cf9a5448debcf10ce3301d1d39 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d20217cf9a5448debcf10ce3301d1d39 2023-04-01 03:58:22,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d20217cf9a5448debcf10ce3301d1d39, entries=50, sequenceid=374, filesize=11.7 K 2023-04-01 03:58:22,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/B/e5f52bb8463045bdb73907b3a505f3be as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e5f52bb8463045bdb73907b3a505f3be 2023-04-01 03:58:22,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e5f52bb8463045bdb73907b3a505f3be, entries=50, sequenceid=374, filesize=7.1 K 2023-04-01 03:58:22,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/.tmp/C/cff1216dd9054c29af234616449b9b5d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/cff1216dd9054c29af234616449b9b5d 2023-04-01 03:58:22,190 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/cff1216dd9054c29af234616449b9b5d, entries=50, sequenceid=374, filesize=7.1 K 2023-04-01 03:58:22,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for c34029273b14edd6bcd250b04ada3f0e in 1267ms, sequenceid=374, compaction requested=false 2023-04-01 03:58:22,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a9e739e122434a944840f89632962d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2] to archive 2023-04-01 03:58:22,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:22,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a9e739e122434a944840f89632962d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/f5a9e739e122434a944840f89632962d 2023-04-01 03:58:22,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/c5fc68d6bd6a49d7b6b84f362282daa0 2023-04-01 03:58:22,200 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/38da0b56c6d14e4b8d2dad210e5e6fc6 2023-04-01 03:58:22,201 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/7cab7d6bfd3d4e64a3a371d15addefc2 2023-04-01 03:58:22,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b6207c8fd4e7446a976b95ad4f730e1f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a] to archive 2023-04-01 03:58:22,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:22,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b6207c8fd4e7446a976b95ad4f730e1f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/b6207c8fd4e7446a976b95ad4f730e1f 2023-04-01 03:58:22,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/6502ce9d9e4a455c881178f94c820d6d 2023-04-01 03:58:22,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/0efca418804d4bccbd27ddc956ea5941 2023-04-01 03:58:22,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/4f16dce0299347d2886abf2a8312e78a 2023-04-01 03:58:22,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/11f02e83bc2b4ae4b344fecda7adc069, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6] to archive 2023-04-01 03:58:22,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:22,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/11f02e83bc2b4ae4b344fecda7adc069 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/11f02e83bc2b4ae4b344fecda7adc069 2023-04-01 03:58:22,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5be1675010d54439bb51729975ecb19c 2023-04-01 03:58:22,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/fb3e5465a6d2434194bd3ea67c4ae196 2023-04-01 03:58:22,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5c4f42da3e104228bf40d2c0799caad6 2023-04-01 03:58:22,229 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits/377.seqid, newMaxSeqId=377, maxSeqId=4 2023-04-01 03:58:22,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e. 2023-04-01 03:58:22,230 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c34029273b14edd6bcd250b04ada3f0e: 2023-04-01 03:58:22,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:22,232 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c34029273b14edd6bcd250b04ada3f0e, regionState=CLOSED 2023-04-01 03:58:22,232 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321502231"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321502231"}]},"ts":"1680321502231"} 2023-04-01 03:58:22,234 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-01 03:58:22,234 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure c34029273b14edd6bcd250b04ada3f0e, server=jenkins-hbase20.apache.org,36251,1680321375816 in 1.4630 sec 2023-04-01 03:58:22,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-01 03:58:22,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c34029273b14edd6bcd250b04ada3f0e, UNASSIGN in 1.4670 sec 2023-04-01 03:58:22,236 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321502235"}]},"ts":"1680321502235"} 2023-04-01 03:58:22,236 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:58:22,238 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:58:22,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4910 sec 2023-04-01 03:58:22,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 03:58:22,858 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-01 03:58:22,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:58:22,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,861 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-01 03:58:22,861 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,874 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:22,876 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits] 2023-04-01 03:58:22,881 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/54efed2278f144868bb37d316e5c52a1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/54efed2278f144868bb37d316e5c52a1 2023-04-01 03:58:22,883 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d20217cf9a5448debcf10ce3301d1d39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/A/d20217cf9a5448debcf10ce3301d1d39 2023-04-01 03:58:22,889 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/85d88184876a4d9aa72e18f7554a1b51 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/85d88184876a4d9aa72e18f7554a1b51 2023-04-01 03:58:22,891 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e5f52bb8463045bdb73907b3a505f3be to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/B/e5f52bb8463045bdb73907b3a505f3be 2023-04-01 03:58:22,894 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5cd1d93eb4d046228b39c3d7a2066f7c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/5cd1d93eb4d046228b39c3d7a2066f7c 2023-04-01 03:58:22,896 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/cff1216dd9054c29af234616449b9b5d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/C/cff1216dd9054c29af234616449b9b5d 2023-04-01 03:58:22,899 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits/377.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e/recovered.edits/377.seqid 2023-04-01 03:58:22,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/c34029273b14edd6bcd250b04ada3f0e 2023-04-01 03:58:22,902 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:58:22,902 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:58:22,903 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 03:58:22,907 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040108c5d71058ea4d1fb56eb92859610e14 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040108c5d71058ea4d1fb56eb92859610e14 2023-04-01 03:58:22,908 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011c2bf0d9bb1246f582dc131b12d95cbe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011c2bf0d9bb1246f582dc131b12d95cbe 2023-04-01 03:58:22,909 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011e5d4c05f6eb418a93e3b6c60467d020 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011e5d4c05f6eb418a93e3b6c60467d020 2023-04-01 03:58:22,911 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012d589edb5cc0493c9b9b5fefdbcb7e27 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012d589edb5cc0493c9b9b5fefdbcb7e27 2023-04-01 03:58:22,912 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb0709485344995a4e8c8b867cc541f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb0709485344995a4e8c8b867cc541f 2023-04-01 03:58:22,913 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401351c40484a1f4a9c90c6eb8dd8e2eb39 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401351c40484a1f4a9c90c6eb8dd8e2eb39 2023-04-01 03:58:22,915 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040137760327e7d94d2b838861df5952d7ac to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040137760327e7d94d2b838861df5952d7ac 2023-04-01 03:58:22,916 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013a2bb2cb561d43c9bb59b286d0ea8414 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013a2bb2cb561d43c9bb59b286d0ea8414 2023-04-01 03:58:22,918 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014723f65d981f482d8b7cadf6d10ded8f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014723f65d981f482d8b7cadf6d10ded8f 2023-04-01 03:58:22,919 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040154add8aec618401780f9cd86af34ae54 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040154add8aec618401780f9cd86af34ae54 2023-04-01 03:58:22,920 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017eafd24583d54fd5bd69fa02593385d0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017eafd24583d54fd5bd69fa02593385d0 2023-04-01 03:58:22,922 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181e2359846ab46a99a2de3af4123edc3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181e2359846ab46a99a2de3af4123edc3 2023-04-01 03:58:22,923 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018384750151e64f899fcc5dcf283f9472 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018384750151e64f899fcc5dcf283f9472 2023-04-01 03:58:22,925 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018ba167ee19264afbbdfbedaeec9ee304 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018ba167ee19264afbbdfbedaeec9ee304 2023-04-01 03:58:22,926 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a07ad9c35e854febbb7a7eec426ae58d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a07ad9c35e854febbb7a7eec426ae58d 2023-04-01 03:58:22,927 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a08f6897812b46ccaf075b3b66bdd4f0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a08f6897812b46ccaf075b3b66bdd4f0 2023-04-01 03:58:22,928 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a9969d7118b04b399e30ca9eeab4fef5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a9969d7118b04b399e30ca9eeab4fef5 2023-04-01 03:58:22,930 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aa0ab99fa08941fbb73a6392f11217d1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aa0ab99fa08941fbb73a6392f11217d1 2023-04-01 03:58:22,931 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ac26be857ff942a49b865adf3389a1a8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ac26be857ff942a49b865adf3389a1a8 2023-04-01 03:58:22,932 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba23674eae024c5e87716cc6bf298b24 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba23674eae024c5e87716cc6bf298b24 2023-04-01 03:58:22,933 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d6b5d251c9f3413f8861681703fb7a81 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d6b5d251c9f3413f8861681703fb7a81 2023-04-01 03:58:22,934 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f73429369696419d9a6ab38a2f637504 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f73429369696419d9a6ab38a2f637504 2023-04-01 03:58:22,935 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:58:22,937 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,938 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:58:22,941 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:58:22,942 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,942 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:58:22,942 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321502942"}]},"ts":"9223372036854775807"} 2023-04-01 03:58:22,944 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:58:22,944 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c34029273b14edd6bcd250b04ada3f0e, NAME => 'TestAcidGuarantees,,1680321473137.c34029273b14edd6bcd250b04ada3f0e.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:58:22,944 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:58:22,944 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321502944"}]},"ts":"9223372036854775807"} 2023-04-01 03:58:22,946 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:58:22,947 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,948 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 89 msec 2023-04-01 03:58:22,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-01 03:58:22,962 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-01 03:58:22,977 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=230 (was 224) - Thread LEAK? -, OpenFileDescriptor=532 (was 547), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=606 (was 476) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=8129 (was 8863) 2023-04-01 03:58:22,990 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230, OpenFileDescriptor=532, MaxFileDescriptor=60000, SystemLoadAverage=606, ProcessCount=167, AvailableMemoryMB=8128 2023-04-01 03:58:22,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:58:22,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:58:22,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:22,995 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:58:22,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-01 03:58:22,996 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:58:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 03:58:22,999 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:22,999 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 empty. 2023-04-01 03:58:22,999 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:22,999 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:58:23,010 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:58:23,011 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 64d06c34ee69fd6096a8db67850840d6, NAME => 'TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:58:23,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 03:58:23,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 64d06c34ee69fd6096a8db67850840d6, disabling compactions & flushes 2023-04-01 03:58:23,430 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. after waiting 0 ms 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,430 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,430 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:23,432 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:58:23,433 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321503433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321503433"}]},"ts":"1680321503433"} 2023-04-01 03:58:23,434 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:58:23,435 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:58:23,435 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321503435"}]},"ts":"1680321503435"} 2023-04-01 03:58:23,436 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:58:23,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, ASSIGN}] 2023-04-01 03:58:23,441 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, ASSIGN 2023-04-01 03:58:23,443 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:58:23,595 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=64d06c34ee69fd6096a8db67850840d6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:23,595 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321503595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321503595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321503595"}]},"ts":"1680321503595"} 2023-04-01 03:58:23,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 03:58:23,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 64d06c34ee69fd6096a8db67850840d6, NAME => 'TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:58:23,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:58:23,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,752 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,753 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A 2023-04-01 03:58:23,753 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A 2023-04-01 03:58:23,753 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:23,754 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64d06c34ee69fd6096a8db67850840d6 columnFamilyName A 2023-04-01 03:58:23,754 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(310): Store=64d06c34ee69fd6096a8db67850840d6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:23,754 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,755 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B 2023-04-01 03:58:23,755 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B 2023-04-01 03:58:23,755 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:23,756 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64d06c34ee69fd6096a8db67850840d6 columnFamilyName B 2023-04-01 03:58:23,756 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(310): Store=64d06c34ee69fd6096a8db67850840d6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:23,756 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,757 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C 2023-04-01 03:58:23,757 DEBUG [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C 2023-04-01 03:58:23,757 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:23,757 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64d06c34ee69fd6096a8db67850840d6 columnFamilyName C 2023-04-01 03:58:23,758 INFO [StoreOpener-64d06c34ee69fd6096a8db67850840d6-1] regionserver.HStore(310): Store=64d06c34ee69fd6096a8db67850840d6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:23,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:58:23,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:23,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:58:23,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 64d06c34ee69fd6096a8db67850840d6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70585644, jitterRate=0.051808059215545654}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:58:23,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:23,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., pid=49, masterSystemTime=1680321503747 2023-04-01 03:58:23,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:23,766 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=64d06c34ee69fd6096a8db67850840d6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:23,766 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321503766"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321503766"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321503766"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321503766"}]},"ts":"1680321503766"} 2023-04-01 03:58:23,768 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-01 03:58:23,769 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 in 171 msec 2023-04-01 03:58:23,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-01 03:58:23,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, ASSIGN in 330 msec 2023-04-01 03:58:23,771 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:58:23,771 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321503771"}]},"ts":"1680321503771"} 2023-04-01 03:58:23,772 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:58:23,779 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:58:23,787 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 786 msec 2023-04-01 03:58:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 03:58:24,100 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-01 03:58:24,101 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x1262233b to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,115 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@737006c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,116 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,119 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,120 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:58:24,124 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40808, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:58:24,125 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x2beab5e2 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,134 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2812305f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,135 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x495719a9 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,142 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@376ce0c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,143 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6eae89d2 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,150 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f455fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,151 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6cec010c to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,184 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7696e55a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,184 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x73e65fce to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,209 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f8ed4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,210 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x73892d82 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,234 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a282700, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,234 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x6cee7d6e to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,259 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5631b84c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,259 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5f8cb3 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,267 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40c58f3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,268 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x22e3658a to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,277 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a4f71, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,278 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x1b543524 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:24,292 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22a2dcf1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:24,302 DEBUG [hconnection-0x5ea330ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,321 DEBUG [hconnection-0x75c6e852-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,321 DEBUG [hconnection-0x4cf04af3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,322 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:24,322 DEBUG [hconnection-0x2d0bc74d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,325 DEBUG [hconnection-0x6c5d50bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,326 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,326 DEBUG [hconnection-0x48ff59fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,334 DEBUG [hconnection-0x23d437fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,336 DEBUG [hconnection-0x353a9009-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,337 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,345 DEBUG [hconnection-0x57e67624-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,345 DEBUG [hconnection-0x5492cc13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:24,348 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,349 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,349 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,349 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,350 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,352 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,353 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:24,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:24,361 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:24,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:24,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:24,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:24,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:24,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:24,367 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:24,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:24,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:24,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:24,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:24,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:24,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:24,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:24,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:24,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:24,378 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:24,378 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:24,379 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-01 03:58:24,410 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:24,410 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:24,410 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:24,410 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:24,410 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:24,411 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:24,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:24,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680321504427, max timestamp=1680321504438Num uniques 859; 2023-04-01 03:58:24,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 03:58:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680321504427, max timestamp=1680321504438Num uniques 859; 2023-04-01 03:58:24,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 03:58:24,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680321504427, max timestamp=1680321504438Num uniques 859; 2023-04-01 03:58:24,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 03:58:24,457 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/a51984d27f7c4d348eae9c8ee006fb29 2023-04-01 03:58:24,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321564457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321564458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321564459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321564462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321564459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321564563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321564570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321564570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321564571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321564574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321564764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321564777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321564777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321564777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321564778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:24,895 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b0355dae33eb4c4b9898ccb860e07135 2023-04-01 03:58:25,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321565067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321565084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321565085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321565085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321565085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,327 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/012a27f84f084531903b07a54dc48feb 2023-04-01 03:58:25,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/a51984d27f7c4d348eae9c8ee006fb29 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29 2023-04-01 03:58:25,333 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29, entries=100, sequenceid=10, filesize=9.3 K 2023-04-01 03:58:25,334 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b0355dae33eb4c4b9898ccb860e07135 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135 2023-04-01 03:58:25,336 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135, entries=100, sequenceid=10, filesize=9.3 K 2023-04-01 03:58:25,337 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/012a27f84f084531903b07a54dc48feb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb 2023-04-01 03:58:25,339 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb, entries=100, sequenceid=10, filesize=9.3 K 2023-04-01 03:58:25,340 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for 64d06c34ee69fd6096a8db67850840d6 in 961ms, sequenceid=10, compaction requested=false 2023-04-01 03:58:25,340 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:25,340 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:25,340 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:25,340 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:25,340 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,340 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:25,340 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:25,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:25,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:25,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:25,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1445477e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:25,344 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:25,344 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:25,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:25,348 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:25,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:25,350 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:25,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:25,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:25,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:25,350 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:25,366 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,367 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:25,367 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:25,367 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:25,368 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:25,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:25,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:25,370 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:25,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:25,370 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:25,370 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,371 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:25,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:25,371 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:25,471 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:25,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:25,483 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:25,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:25,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:25,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:25,484 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,484 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:25,485 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:25,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:25,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:25,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:25,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:25,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:25,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:25,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:25,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:25,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:25,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:25,486 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:25,486 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:25,486 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:25,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:25,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:25,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321565587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321565588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321565589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321565590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321565592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321565690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:25,894 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/93ffe706292a403b9044ff6aacd278cd 2023-04-01 03:58:25,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:25,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321565898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321566202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,303 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/93ed01385f3c431989258c593efe85c4 2023-04-01 03:58:26,315 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/d5cb16181f004901a33588cf12cef4e9 2023-04-01 03:58:26,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/93ffe706292a403b9044ff6aacd278cd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd 2023-04-01 03:58:26,320 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd, entries=150, sequenceid=46, filesize=11.6 K 2023-04-01 03:58:26,321 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/93ed01385f3c431989258c593efe85c4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4 2023-04-01 03:58:26,323 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4, entries=150, sequenceid=46, filesize=11.6 K 2023-04-01 03:58:26,325 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/d5cb16181f004901a33588cf12cef4e9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9 2023-04-01 03:58:26,329 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9, entries=150, sequenceid=46, filesize=11.6 K 2023-04-01 03:58:26,330 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 64d06c34ee69fd6096a8db67850840d6 in 844ms, sequenceid=46, compaction requested=false 2023-04-01 03:58:26,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:26,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,331 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:26,331 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,333 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,333 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:26,333 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:26,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:26,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590f6828[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:26,336 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:26,336 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:26,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:26,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:26,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:26,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:26,337 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:26,337 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:26,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:26,342 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:26,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:26,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:26,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:26,342 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:26,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:26,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:26,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:26,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,349 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:26,349 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:26,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:26,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:26,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:26,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:26,349 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:26,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:26,349 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:26,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,450 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:26,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:26,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:26,454 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:26,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:26,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:26,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:26,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:26,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,456 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:26,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:26,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:26,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:26,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:26,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:26,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:26,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:26,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:26,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:26,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:26,458 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:26,458 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:26,458 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:26,458 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 03:58:26,458 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:26,458 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:26,458 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:26,459 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:26,459 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:26,459 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:26,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321506612, max timestamp=1680321506624Num uniques 859; 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321506612, max timestamp=1680321506624Num uniques 859; 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321506612, max timestamp=1680321506624Num uniques 859; 2023-04-01 03:58:26,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:26,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680321506667, max timestamp=1680321506677Num uniques 1068; 2023-04-01 03:58:26,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680321506667, max timestamp=1680321506677Num uniques 1068; 2023-04-01 03:58:26,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 03:58:26,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 03:58:26,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680321506667, max timestamp=1680321506677Num uniques 1068; 2023-04-01 03:58:26,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 03:58:26,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321566708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680321506696, max timestamp=1680321506717Num uniques 1177; 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680321506696, max timestamp=1680321506717Num uniques 1177; 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680321506696, max timestamp=1680321506717Num uniques 1177; 2023-04-01 03:58:26,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-01 03:58:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321566746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321566746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321566747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321566751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321566819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321566854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321566854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321566857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:26,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321566857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:26,874 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/c6013b9a54e642b0969aaa9f7a2d2e11 2023-04-01 03:58:26,887 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b8a3f3d8ac28491288b39b79297a134a 2023-04-01 03:58:27,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321567025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321567062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321567062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321567063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321567064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,301 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/44c8e9fb0b9e496bbfa6f998d98a40fe 2023-04-01 03:58:27,304 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/c6013b9a54e642b0969aaa9f7a2d2e11 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11 2023-04-01 03:58:27,306 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11, entries=150, sequenceid=54, filesize=11.6 K 2023-04-01 03:58:27,310 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b8a3f3d8ac28491288b39b79297a134a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a 2023-04-01 03:58:27,313 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a, entries=150, sequenceid=54, filesize=11.6 K 2023-04-01 03:58:27,314 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/44c8e9fb0b9e496bbfa6f998d98a40fe as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe 2023-04-01 03:58:27,317 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe, entries=150, sequenceid=54, filesize=11.6 K 2023-04-01 03:58:27,317 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 64d06c34ee69fd6096a8db67850840d6 in 859ms, sequenceid=54, compaction requested=true 2023-04-01 03:58:27,317 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:27,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:27,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:27,327 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,327 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:27,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d322a14[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:27,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:27,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=154.31 KB heapSize=434.39 KB 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:27,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:27,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:27,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:27,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:27,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:27,335 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,335 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:27,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:27,336 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,336 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:27,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:27,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:27,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:27,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:27,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:27,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:27,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,340 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:27,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:27,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:27,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,342 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:27,342 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:27,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321567356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321567365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321567365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321567366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321567367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,442 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:27,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:27,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:27,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:27,446 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:27,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:27,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:27,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:27,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:27,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:27,453 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:27,453 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,453 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:27,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:27,460 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,460 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:27,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:27,460 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:27,460 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:27,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:27,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:27,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:27,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,462 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:27,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337383a5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:27,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:27,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,464 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:27,463 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:27,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:27,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:27,465 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:27,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321567460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:27,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:27,468 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:27,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:27,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:27,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:27,468 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:27,485 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,485 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,485 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,487 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,489 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:27,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,489 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,489 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:27,489 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,589 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:27,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:27,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:27,592 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:27,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:27,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:27,593 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,593 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,595 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:27,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:27,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:27,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:27,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:27,608 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:27,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d334c02[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:27,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:27,609 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,611 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:27,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:27,611 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:27,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,611 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:27,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:27,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:27,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:27,614 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:27,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:27,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:27,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:27,614 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:27,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,616 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,617 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,618 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:27,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:27,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,618 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:27,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,619 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,619 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:27,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,619 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:27,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321567667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,719 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:27,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:27,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:27,734 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:27,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:27,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:27,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,735 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:27,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:27,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:27,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:27,746 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:27,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,750 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,750 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,750 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:27,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5f2ba4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:27,751 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:27,751 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,752 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,752 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:27,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:27,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:27,753 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,753 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:27,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:27,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,753 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,753 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:27,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:27,755 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:27,755 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:27,755 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:27,755 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:27,755 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:27,756 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,756 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,756 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,756 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,758 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,758 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:27,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:27,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:27,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,759 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:27,759 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:27,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/18455d2efbcc4b41b0598046f363a885 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,761 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:58:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,791 WARN [DataStreamer for file /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/84706160fb1c4d29a12c79a94083728c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 03:58:27,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/84706160fb1c4d29a12c79a94083728c 2023-04-01 03:58:27,859 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:27,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:27,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:27,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:27,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:27,863 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:27,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:27,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:27,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,863 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,864 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,864 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:27,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:27,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321567870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321567871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321567871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:27,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321567874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:27,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:27,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:27,882 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:27,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,901 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:27,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391c6d1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:27,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:27,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,904 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:27,904 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:27,904 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:27,904 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:27,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,905 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:27,905 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,905 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:27,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:27,907 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:27,907 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:27,907 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:27,907 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:27,907 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:27,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:27,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:27,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,912 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:27,912 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:27,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:27,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:27,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:27,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:27,913 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:27,913 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:27,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:27,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50526 deadline: 1680321567970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,014 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:28,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:28,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:28,016 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:28,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:28,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:28,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,016 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,017 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:28,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:28,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:28,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,018 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:28,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:28,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:28,024 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,024 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:28,024 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:28,024 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:28,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:28,025 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:28,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5013ad[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:28,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:28,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,027 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:28,027 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,027 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:28,028 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,028 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:28,029 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,029 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:28,032 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:28,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:28,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:28,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:28,032 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:28,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:28,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,033 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:28,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:28,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:28,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:28,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:28,036 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,037 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:28,037 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:28,037 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,037 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,037 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,037 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:28,137 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:28,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:28,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:28,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:28,144 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:28,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:28,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:28,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,146 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:28,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:28,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:28,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:28,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:28,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:28,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:28,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:28,147 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,149 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:28,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b941979[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:28,152 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:28,152 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:28,153 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,153 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:28,154 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:28,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:28,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:28,156 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:28,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:28,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:28,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:28,156 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:28,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:28,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:28,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:28,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:28,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:28,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:28,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:28,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:28,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:28,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:28,163 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:28,163 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:28,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:28,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/2fe8fd729e8b407ba2a463d169ab6ee2 2023-04-01 03:58:28,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/18455d2efbcc4b41b0598046f363a885 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885 2023-04-01 03:58:28,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885, entries=200, sequenceid=126, filesize=13.9 K 2023-04-01 03:58:28,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/84706160fb1c4d29a12c79a94083728c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c 2023-04-01 03:58:28,263 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:28,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:28,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c, entries=150, sequenceid=126, filesize=11.6 K 2023-04-01 03:58:28,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:28,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:28,266 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:28,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:28,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:28,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/2fe8fd729e8b407ba2a463d169ab6ee2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2 2023-04-01 03:58:28,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2, entries=150, sequenceid=126, filesize=11.6 K 2023-04-01 03:58:28,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 64d06c34ee69fd6096a8db67850840d6 in 936ms, sequenceid=126, compaction requested=true 2023-04-01 03:58:28,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:28,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:28,269 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:28,269 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:28,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:28,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:28,270 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:28,270 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:28,270 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:28,270 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,270 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:28,270 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=46.4 K 2023-04-01 03:58:28,270 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,270 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=44.1 K 2023-04-01 03:58:28,270 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting a51984d27f7c4d348eae9c8ee006fb29, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321504369 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b0355dae33eb4c4b9898ccb860e07135, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321504369 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 93ffe706292a403b9044ff6aacd278cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680321504449 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 93ed01385f3c431989258c593efe85c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680321504449 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c6013b9a54e642b0969aaa9f7a2d2e11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680321505582 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b8a3f3d8ac28491288b39b79297a134a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680321505582 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 18455d2efbcc4b41b0598046f363a885, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:28,271 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 84706160fb1c4d29a12c79a94083728c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:28,272 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:28,272 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:28,272 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,272 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:28,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:28,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:28,279 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:28,280 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#387 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:28,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:28,280 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:28,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,280 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:28,280 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:28,281 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:28,311 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/bf3dac1592334c22a130eb5e5869c378 2023-04-01 03:58:28,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:28,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321508518, max timestamp=1680321508569Num uniques 859; 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321508518, max timestamp=1680321508569Num uniques 859; 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321508518, max timestamp=1680321508569Num uniques 859; 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:28,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:28,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680321508625, max timestamp=1680321508661Num uniques 1018; 2023-04-01 03:58:28,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 03:58:28,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680321508625, max timestamp=1680321508661Num uniques 1018; 2023-04-01 03:58:28,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 03:58:28,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680321508625, max timestamp=1680321508661Num uniques 1018; 2023-04-01 03:58:28,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 03:58:28,703 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/f911af8a28604018afb17984d6216511 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f911af8a28604018afb17984d6216511 2023-04-01 03:58:28,706 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into f911af8a28604018afb17984d6216511(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:28,706 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:28,706 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=12, startTime=1680321508269; duration=0sec 2023-04-01 03:58:28,707 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:28,707 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:28,708 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:28,708 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:28,708 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:28,708 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=44.1 K 2023-04-01 03:58:28,708 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 012a27f84f084531903b07a54dc48feb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680321504369 2023-04-01 03:58:28,709 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d5cb16181f004901a33588cf12cef4e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680321504449 2023-04-01 03:58:28,709 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 44c8e9fb0b9e496bbfa6f998d98a40fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680321505582 2023-04-01 03:58:28,709 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2fe8fd729e8b407ba2a463d169ab6ee2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:28,711 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/db1b7f0f1a644f74a51ea3373a26e049 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/db1b7f0f1a644f74a51ea3373a26e049 2023-04-01 03:58:28,714 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:28,714 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into db1b7f0f1a644f74a51ea3373a26e049(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:28,714 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:28,714 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=12, startTime=1680321508269; duration=0sec 2023-04-01 03:58:28,714 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:28,730 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/1f709ccc15d7475e89b8e014476d896a 2023-04-01 03:58:28,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680321508739, max timestamp=1680321508764Num uniques 1077; 2023-04-01 03:58:28,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-01 03:58:28,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680321508739, max timestamp=1680321508764Num uniques 1077; 2023-04-01 03:58:28,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-01 03:58:28,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680321508739, max timestamp=1680321508764Num uniques 1077; 2023-04-01 03:58:28,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-01 03:58:28,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680321508823, max timestamp=1680321508847Num uniques 1036; 2023-04-01 03:58:28,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-01 03:58:28,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680321508823, max timestamp=1680321508847Num uniques 1036; 2023-04-01 03:58:28,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-01 03:58:28,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680321508823, max timestamp=1680321508847Num uniques 1036; 2023-04-01 03:58:28,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-01 03:58:28,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680321508888, max timestamp=1680321508896Num uniques 1045; 2023-04-01 03:58:28,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-01 03:58:28,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680321508888, max timestamp=1680321508896Num uniques 1045; 2023-04-01 03:58:28,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-01 03:58:28,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680321508888, max timestamp=1680321508896Num uniques 1045; 2023-04-01 03:58:28,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-01 03:58:28,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:28,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321568914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321568917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:28,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321568920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:28,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321568920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:28,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:28,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321568921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321569021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321569021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321569023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321569026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321569026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,120 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/224cb3cc0f5946d195c21d0ec5002e85 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/224cb3cc0f5946d195c21d0ec5002e85 2023-04-01 03:58:29,124 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 224cb3cc0f5946d195c21d0ec5002e85(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:29,124 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:29,124 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=12, startTime=1680321508269; duration=0sec 2023-04-01 03:58:29,124 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:29,138 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/cb8d69d64d8a4245be8db4a3c3a17933 2023-04-01 03:58:29,144 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/bf3dac1592334c22a130eb5e5869c378 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378 2023-04-01 03:58:29,147 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378, entries=100, sequenceid=133, filesize=9.4 K 2023-04-01 03:58:29,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/1f709ccc15d7475e89b8e014476d896a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a 2023-04-01 03:58:29,150 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a, entries=100, sequenceid=133, filesize=9.4 K 2023-04-01 03:58:29,150 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/cb8d69d64d8a4245be8db4a3c3a17933 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933 2023-04-01 03:58:29,153 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933, entries=100, sequenceid=133, filesize=9.4 K 2023-04-01 03:58:29,153 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=167.72 KB/171750 for 64d06c34ee69fd6096a8db67850840d6 in 873ms, sequenceid=133, compaction requested=false 2023-04-01 03:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:29,154 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:29,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d4007a4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:29,156 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,156 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,157 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:29,157 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:29,157 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:29,157 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:29,158 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:29,161 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:29,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:29,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:29,161 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:29,161 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:29,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,162 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:29,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:29,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:29,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,164 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,165 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,165 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,165 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=174.43 KB heapSize=483.09 KB 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:29,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 03:58:29,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:29,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321569232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321569232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321569233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321569234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321569234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/ca4a69a702974413a06f4754b3443991 2023-04-01 03:58:29,265 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:29,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:29,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:29,267 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:29,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:29,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:29,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,268 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:29,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:29,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:29,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:29,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:29,269 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:29,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,271 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60eae393[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:29,271 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:29,271 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,272 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,272 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:29,272 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,272 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:29,272 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:29,272 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:29,273 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:29,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:29,287 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:29,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:29,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:29,287 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:29,287 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:29,288 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,288 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,288 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,289 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:29,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,293 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:29,293 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,293 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,294 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,294 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:29,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321569334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321569338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321569340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,394 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:29,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:29,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:29,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:29,396 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:29,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:29,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:29,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:29,426 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:29,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:29,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:29,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:29,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:29,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:29,429 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:29,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:29,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:29,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c8b5958[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:29,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:29,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,433 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:29,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:29,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:29,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:29,434 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:29,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:29,436 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:29,436 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:29,436 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:29,436 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:29,436 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:29,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,438 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:29,442 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:29,443 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:29,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:29,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,543 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:29,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:29,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321569541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321569542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321569542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:29,545 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:29,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:29,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:29,545 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50482 deadline: 1680321569543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:29,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:29,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:29,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:29,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:29,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:29,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:29,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:29,552 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:29,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:29,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321569548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:29,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57cfce61[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:29,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:29,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:29,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:29,555 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,555 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:29,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:29,555 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,555 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,556 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:29,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:29,557 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:29,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:29,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:29,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:29,557 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:29,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:29,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,558 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:29,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:29,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,561 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:29,561 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:29,561 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:29,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:29,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b110582d6a5640d58a4909a7e0c64541 2023-04-01 03:58:29,661 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:29,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:29,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:29,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/e962b24b0a214dc1b54e8ca4755b16f9 2023-04-01 03:58:29,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:29,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:29,664 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:29,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:29,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:29,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/ca4a69a702974413a06f4754b3443991 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991 2023-04-01 03:58:29,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991, entries=200, sequenceid=243, filesize=14.1 K 2023-04-01 03:58:29,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b110582d6a5640d58a4909a7e0c64541 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541 2023-04-01 03:58:29,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541, entries=150, sequenceid=243, filesize=11.8 K 2023-04-01 03:58:29,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/e962b24b0a214dc1b54e8ca4755b16f9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9 2023-04-01 03:58:29,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9, entries=150, sequenceid=243, filesize=11.8 K 2023-04-01 03:58:29,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~483.05 KB/494640, currentSize=13.42 KB/13740 for 64d06c34ee69fd6096a8db67850840d6 in 445ms, sequenceid=243, compaction requested=true 2023-04-01 03:58:29,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:29,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:29,675 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:29,675 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:29,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:29,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:29,675 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36092 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:29,676 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,676 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,676 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f911af8a28604018afb17984d6216511, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=35.2 K 2023-04-01 03:58:29,676 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/db1b7f0f1a644f74a51ea3373a26e049, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=32.9 K 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f911af8a28604018afb17984d6216511, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting db1b7f0f1a644f74a51ea3373a26e049, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting bf3dac1592334c22a130eb5e5869c378, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680321507343 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1f709ccc15d7475e89b8e014476d896a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680321507343 2023-04-01 03:58:29,676 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ca4a69a702974413a06f4754b3443991, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:29,677 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b110582d6a5640d58a4909a7e0c64541, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:29,677 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:29,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:29,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:29,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:29,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:29,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:29,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:29,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:29,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:29,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:29,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:29,679 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:29,679 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:29,679 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:29,680 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:29,681 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:29,684 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:29,697 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/d38a2fdbcb874fdb8b9c8361b506d465 2023-04-01 03:58:29,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:29,855 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:29,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680321509865, max timestamp=1680321509876Num uniques 859; 2023-04-01 03:58:29,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:58:29,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680321509865, max timestamp=1680321509876Num uniques 859; 2023-04-01 03:58:29,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680321509865, max timestamp=1680321509876Num uniques 859; 2023-04-01 03:58:29,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:58:29,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:58:29,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321509899, max timestamp=1680321509910Num uniques 1118; 2023-04-01 03:58:29,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-01 03:58:29,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321509899, max timestamp=1680321509910Num uniques 1118; 2023-04-01 03:58:29,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-01 03:58:29,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680321509899, max timestamp=1680321509910Num uniques 1118; 2023-04-01 03:58:29,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680321509931, max timestamp=1680321509968Num uniques 1027; 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680321509931, max timestamp=1680321509968Num uniques 1027; 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680321509931, max timestamp=1680321509968Num uniques 1027; 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-01 03:58:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680321509990, max timestamp=1680321510005Num uniques 1236; 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680321509990, max timestamp=1680321510005Num uniques 1236; 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=13096, suffix heap size=156512,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=286 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680321509990, max timestamp=1680321510005Num uniques 1236; 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=13096, suffix heap size=156512,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=286 2023-04-01 03:58:30,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=13096, suffix heap size=156512,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=286 2023-04-01 03:58:30,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321510043, max timestamp=1680321510054Num uniques 1145; 2023-04-01 03:58:30,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-01 03:58:30,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321510043, max timestamp=1680321510054Num uniques 1145; 2023-04-01 03:58:30,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-01 03:58:30,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680321510043, max timestamp=1680321510054Num uniques 1145; 2023-04-01 03:58:30,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-01 03:58:30,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680321510065, max timestamp=1680321510078Num uniques 1154; 2023-04-01 03:58:30,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=16210, suffix heap size=147984,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=354 2023-04-01 03:58:30,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680321510065, max timestamp=1680321510078Num uniques 1154; 2023-04-01 03:58:30,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=16210, suffix heap size=147984,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=354 2023-04-01 03:58:30,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680321510065, max timestamp=1680321510078Num uniques 1154; 2023-04-01 03:58:30,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-01 03:58:30,093 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/99e1dacbc0364a4abba45690fdd165c0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/99e1dacbc0364a4abba45690fdd165c0 2023-04-01 03:58:30,097 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 99e1dacbc0364a4abba45690fdd165c0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:30,097 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:30,097 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321509675; duration=0sec 2023-04-01 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:30,102 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:30,102 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:30,102 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,102 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/224cb3cc0f5946d195c21d0ec5002e85, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=32.9 K 2023-04-01 03:58:30,103 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 224cb3cc0f5946d195c21d0ec5002e85, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680321506744 2023-04-01 03:58:30,103 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting cb8d69d64d8a4245be8db4a3c3a17933, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680321507343 2023-04-01 03:58:30,103 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e962b24b0a214dc1b54e8ca4755b16f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:30,106 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/614425eb901b433a9ff8104993841c03 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/614425eb901b433a9ff8104993841c03 2023-04-01 03:58:30,111 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:30,111 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into 614425eb901b433a9ff8104993841c03(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:30,112 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:30,112 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321509675; duration=0sec 2023-04-01 03:58:30,112 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:30,116 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/adeba511fd5c4f30829a79f33d77951f 2023-04-01 03:58:30,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50512 deadline: 1680321570111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321570111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50506 deadline: 1680321570113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321510109, max timestamp=1680321510118Num uniques 1063; 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321510109, max timestamp=1680321510118Num uniques 1213; 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=7458, suffix heap size=154120,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=163 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680321510109, max timestamp=1680321510118Num uniques 1213; 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=7458, suffix heap size=154120,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=163 2023-04-01 03:58:30,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7868, totalHeapSize=18128, min timestamp=1680321510169, max timestamp=1680321510189Num uniques 1022; 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46798, new segment data size=7868, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7867, totalHeapSize=18128, min timestamp=1680321510169, max timestamp=1680321510189Num uniques 1022; 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46797, new segment data size=7867, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7862, totalHeapSize=18128, min timestamp=1680321510169, max timestamp=1680321510189Num uniques 1022; 2023-04-01 03:58:30,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46792, new segment data size=7862, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-01 03:58:30,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321510231, max timestamp=1680321510241Num uniques 1031; 2023-04-01 03:58:30,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321510231, max timestamp=1680321510241Num uniques 1031; 2023-04-01 03:58:30,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:58:30,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:58:30,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680321510231, max timestamp=1680321510241Num uniques 1031; 2023-04-01 03:58:30,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-01 03:58:30,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680321510256, max timestamp=1680321510289Num uniques 1040; 2023-04-01 03:58:30,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-01 03:58:30,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680321510256, max timestamp=1680321510289Num uniques 1040; 2023-04-01 03:58:30,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-01 03:58:30,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680321510256, max timestamp=1680321510289Num uniques 1040; 2023-04-01 03:58:30,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680321510312, max timestamp=1680321510320Num uniques 1099; 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680321510312, max timestamp=1680321510320Num uniques 1099; 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680321510312, max timestamp=1680321510320Num uniques 1099; 2023-04-01 03:58:30,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 03:58:30,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680321510345, max timestamp=1680321510372Num uniques 1108; 2023-04-01 03:58:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680321510345, max timestamp=1680321510372Num uniques 1108; 2023-04-01 03:58:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-01 03:58:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-01 03:58:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680321510345, max timestamp=1680321510372Num uniques 1108; 2023-04-01 03:58:30,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680321510388, max timestamp=1680321510402Num uniques 1067; 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680321510388, max timestamp=1680321510402Num uniques 1067; 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680321510388, max timestamp=1680321510402Num uniques 1067; 2023-04-01 03:58:30,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680321510419, max timestamp=1680321510427Num uniques 1126; 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680321510419, max timestamp=1680321510427Num uniques 1126; 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680321510419, max timestamp=1680321510427Num uniques 1126; 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-01 03:58:30,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-01 03:58:30,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680321510443, max timestamp=1680321510461Num uniques 1135; 2023-04-01 03:58:30,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-01 03:58:30,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680321510443, max timestamp=1680321510461Num uniques 1135; 2023-04-01 03:58:30,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-01 03:58:30,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680321510450, max timestamp=1680321510461Num uniques 1135; 2023-04-01 03:58:30,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680321510481, max timestamp=1680321510504Num uniques 1194; 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680321510481, max timestamp=1680321510504Num uniques 1194; 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=8884, suffix heap size=152144,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=194 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=8884, suffix heap size=152144,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=194 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680321510481, max timestamp=1680321510504Num uniques 1044; 2023-04-01 03:58:30,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-01 03:58:30,528 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/6fdebfdd25394b538412e25533493a1e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/6fdebfdd25394b538412e25533493a1e 2023-04-01 03:58:30,531 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 6fdebfdd25394b538412e25533493a1e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:30,531 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:30,532 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321509675; duration=0sec 2023-04-01 03:58:30,532 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:30,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680321510529, max timestamp=1680321510538Num uniques 1053; 2023-04-01 03:58:30,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-01 03:58:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680321510529, max timestamp=1680321510538Num uniques 1053; 2023-04-01 03:58:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-01 03:58:30,546 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/40caf61bf5404fd7ba043acf5d197449 2023-04-01 03:58:30,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680321510529, max timestamp=1680321510538Num uniques 1053; 2023-04-01 03:58:30,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-01 03:58:30,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/d38a2fdbcb874fdb8b9c8361b506d465 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465 2023-04-01 03:58:30,555 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465, entries=50, sequenceid=248, filesize=7.1 K 2023-04-01 03:58:30,556 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/adeba511fd5c4f30829a79f33d77951f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f 2023-04-01 03:58:30,564 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f, entries=50, sequenceid=248, filesize=7.1 K 2023-04-01 03:58:30,564 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/40caf61bf5404fd7ba043acf5d197449 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449 2023-04-01 03:58:30,573 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449, entries=50, sequenceid=248, filesize=7.1 K 2023-04-01 03:58:30,574 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=127.47 KB/130530 for 64d06c34ee69fd6096a8db67850840d6 in 895ms, sequenceid=248, compaction requested=false 2023-04-01 03:58:30,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:30,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:30,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:30,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:30,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=140.89 KB heapSize=400.78 KB 2023-04-01 03:58:30,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:30,575 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:30,576 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1680321510554, max timestamp=1680321510575Num uniques 1062; 2023-04-01 03:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=14282, suffix heap size=138416,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=312 2023-04-01 03:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1680321510554, max timestamp=1680321510575Num uniques 1062; 2023-04-01 03:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680321510566, max timestamp=1680321510575Num uniques 1062; 2023-04-01 03:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:30,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-01 03:58:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=14282, suffix heap size=138416,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=312 2023-04-01 03:58:30,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:30,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3adf8e1b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,579 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:30,579 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,580 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:30,580 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:30,580 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:30,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,581 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,581 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:30,581 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:30,581 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,581 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:30,581 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:30,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:30,584 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:30,584 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:30,584 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:30,584 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:30,585 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:30,586 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,586 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,586 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,590 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,594 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:30,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:30,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:30,594 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:30,594 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:30,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:30,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/bf1d0c65d54c4dc99aedf754d3f1ddf1 2023-04-01 03:58:30,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321510593, max timestamp=1680321510614Num uniques 859; 2023-04-01 03:58:30,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:30,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321510593, max timestamp=1680321510614Num uniques 859; 2023-04-01 03:58:30,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:58:30,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321510593, max timestamp=1680321510614Num uniques 859; 2023-04-01 03:58:30,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:58:30,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/68aed5c09da5484781afa34f433a073e 2023-04-01 03:58:30,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/037d35e9f6f2484bae5337e0823d2503 2023-04-01 03:58:30,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/bf1d0c65d54c4dc99aedf754d3f1ddf1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1 2023-04-01 03:58:30,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1, entries=200, sequenceid=564, filesize=14.3 K 2023-04-01 03:58:30,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/68aed5c09da5484781afa34f433a073e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e 2023-04-01 03:58:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321570648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680321510639, max timestamp=1680321510657Num uniques 1009; 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680321510639, max timestamp=1680321510657Num uniques 1068; 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680321510639, max timestamp=1680321510657Num uniques 1009; 2023-04-01 03:58:30,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-01 03:58:30,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e, entries=150, sequenceid=564, filesize=11.9 K 2023-04-01 03:58:30,667 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/037d35e9f6f2484bae5337e0823d2503 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503 2023-04-01 03:58:30,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503, entries=150, sequenceid=564, filesize=11.9 K 2023-04-01 03:58:30,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~96.38 KB/98688, currentSize=73.80 KB/75570 for 64d06c34ee69fd6096a8db67850840d6 in 96ms, sequenceid=564, compaction requested=true 2023-04-01 03:58:30,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:30,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:30,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:30,671 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:30,671 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:30,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 03:58:30,672 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34152 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:30,672 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31712 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:30,672 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:30,673 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,673 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,673 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/614425eb901b433a9ff8104993841c03, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=33.4 K 2023-04-01 03:58:30,673 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/99e1dacbc0364a4abba45690fdd165c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=31.0 K 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 614425eb901b433a9ff8104993841c03, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 99e1dacbc0364a4abba45690fdd165c0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d38a2fdbcb874fdb8b9c8361b506d465, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680321509232 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting adeba511fd5c4f30829a79f33d77951f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680321509232 2023-04-01 03:58:30,673 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bf1d0c65d54c4dc99aedf754d3f1ddf1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:30,675 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 68aed5c09da5484781afa34f433a073e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:30,678 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=80.51 KB heapSize=215.39 KB 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:30,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:30,685 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:30,687 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:30,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/70b007854d974db3bb9d74526683f619 2023-04-01 03:58:30,694 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:30,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:30,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:30,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321570717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321570718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321570727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321570727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:30,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:30,743 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:30,743 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:30,743 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:30,743 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:30,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,745 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,745 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:30,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:30,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:30,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:30,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:30,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:30,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:30,755 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:30,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:30,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:30,759 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,759 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:30,760 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:30,760 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:30,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:30,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:30,760 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:30,767 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:30,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1e14a8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:30,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:30,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:30,774 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:30,775 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:30,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321570774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:30,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:30,785 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:30,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:30,788 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:30,788 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:30,788 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:30,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:30,789 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:30,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:30,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:30,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:30,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,801 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:30,801 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:30,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:30,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321570828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321570828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321570839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321570839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,901 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:30,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:30,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:30,904 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:30,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:30,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:30,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:30,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:30,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:30,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:30,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:30,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:30,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:30,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:30,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:30,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:30,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:30,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,908 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:30,908 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:30,908 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:30,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:30,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:30,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:30,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:30,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:30,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc9ddca[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:30,912 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:30,912 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:30,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,915 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:30,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:30,917 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:30,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:30,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:30,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:30,917 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:30,919 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,919 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,919 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:30,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:30,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:30,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:30,924 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:30,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:30,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:30,924 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,925 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:30,925 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:30,925 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:30,925 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:30,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:31,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50504 deadline: 1680321570983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,025 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:31,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:31,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:31,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:31,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:31,032 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:31,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:31,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:31,033 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,033 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,034 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:31,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:31,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:31,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:31,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:31,035 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:31,035 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,035 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:31,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:31,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:31,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:31,036 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:31,037 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:31,039 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:31,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dca414[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:31,039 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,041 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:31,042 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:31,042 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:31,042 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:31,043 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321571039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321571040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:31,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:31,046 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:31,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:31,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:31,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:31,047 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:31,048 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,048 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,048 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:31,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,050 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:31,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:31,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:31,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:31,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:31,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:31,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:31,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,052 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:31,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,052 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:31,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:31,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:31,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321571053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321571053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,095 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/253d6d20838c4246a3cba26936087ea2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/253d6d20838c4246a3cba26936087ea2 2023-04-01 03:58:31,099 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into 253d6d20838c4246a3cba26936087ea2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:31,100 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:31,100 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321510671; duration=0sec 2023-04-01 03:58:31,100 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:31,100 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:31,101 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31712 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:31,101 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:31,101 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,101 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/6fdebfdd25394b538412e25533493a1e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=31.0 K 2023-04-01 03:58:31,101 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6fdebfdd25394b538412e25533493a1e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680321508919 2023-04-01 03:58:31,101 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 40caf61bf5404fd7ba043acf5d197449, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680321509232 2023-04-01 03:58:31,102 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 037d35e9f6f2484bae5337e0823d2503, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:31,107 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#409 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:31,134 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/c236152eed164695ac7914cda396dea7 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c236152eed164695ac7914cda396dea7 2023-04-01 03:58:31,140 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into c236152eed164695ac7914cda396dea7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:31,140 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:31,140 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321510671; duration=0sec 2023-04-01 03:58:31,140 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:31,152 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:31,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:31,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:31,156 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/e17f0baef29e4568b44b8251a778d956 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/e17f0baef29e4568b44b8251a778d956 2023-04-01 03:58:31,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:31,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:31,158 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:31,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:31,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:31,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/2d688dce30634d4eb0ed83d4bcfe2a3a 2023-04-01 03:58:31,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:31,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:31,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:31,164 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into e17f0baef29e4568b44b8251a778d956(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:31,164 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:31,164 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321510671; duration=0sec 2023-04-01 03:58:31,164 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:31,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:31,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:31,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:31,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:31,170 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:31,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:31,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:31,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74354b02[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:31,173 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:31,173 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,174 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,174 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:31,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:31,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:31,175 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:31,176 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:31,176 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:31,176 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,176 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:31,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:31,178 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:31,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:31,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:31,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:31,178 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:31,181 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,182 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,182 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:31,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:31,185 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,188 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:31,191 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:31,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/7f1f507b4d094db8b7f8b698b32a054f 2023-04-01 03:58:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:31,192 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:31,192 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:31,192 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:31,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/70b007854d974db3bb9d74526683f619 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619 2023-04-01 03:58:31,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619, entries=200, sequenceid=609, filesize=14.3 K 2023-04-01 03:58:31,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/2d688dce30634d4eb0ed83d4bcfe2a3a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a 2023-04-01 03:58:31,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a, entries=200, sequenceid=609, filesize=14.3 K 2023-04-01 03:58:31,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/7f1f507b4d094db8b7f8b698b32a054f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f 2023-04-01 03:58:31,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f, entries=150, sequenceid=609, filesize=11.9 K 2023-04-01 03:58:31,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~235.27 KB/240912, currentSize=93.93 KB/96180 for 64d06c34ee69fd6096a8db67850840d6 in 534ms, sequenceid=609, compaction requested=false 2023-04-01 03:58:31,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:31,293 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:31,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:31,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:31,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:31,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:31,302 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:31,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:31,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:31,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:31,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:31,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:31,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:31,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:31,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:31,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:31,306 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:31,306 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:31,307 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:31,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:31,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:31,325 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/4feaf550a2a648e3bda7a81d243aa838 2023-04-01 03:58:31,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321571350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321571351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321571361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321571361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321571361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321571462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321571462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321571469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321571670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321571671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321571674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,742 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/449bcbb36fd4450c938a695949ea7033 2023-04-01 03:58:31,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321571868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321571872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321571976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321571980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:31,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:31,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321571980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,164 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/90fef2e9368c4b83a8ace126815b3037 2023-04-01 03:58:32,168 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/4feaf550a2a648e3bda7a81d243aa838 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838 2023-04-01 03:58:32,174 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838, entries=150, sequenceid=629, filesize=11.9 K 2023-04-01 03:58:32,175 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/449bcbb36fd4450c938a695949ea7033 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033 2023-04-01 03:58:32,179 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033, entries=150, sequenceid=629, filesize=11.9 K 2023-04-01 03:58:32,180 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/90fef2e9368c4b83a8ace126815b3037 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037 2023-04-01 03:58:32,187 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037, entries=150, sequenceid=629, filesize=11.9 K 2023-04-01 03:58:32,188 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 64d06c34ee69fd6096a8db67850840d6 in 882ms, sequenceid=629, compaction requested=true 2023-04-01 03:58:32,188 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:32,188 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,190 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:32,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f3ba78[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:32,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:32,193 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,193 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:32,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,194 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:32,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:32,216 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:32,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:32,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:32,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:32,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:32,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:32,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:32,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,221 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,223 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,241 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:32,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:32,242 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,242 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:32,242 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,242 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,342 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:32,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:32,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:32,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:32,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:32,345 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:32,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:32,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:32,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,347 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:32,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:32,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:32,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:32,349 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:32,349 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,349 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:32,359 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:32,359 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:32,359 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,359 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:32,360 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:32,360 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,366 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/5edc4697f58f464f84b37fb7c52a5859 2023-04-01 03:58:32,389 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/cc7496cbd3314a70818ad881c1b1b30f 2023-04-01 03:58:32,406 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/c99a540de9a244588cf26636713adc97 2023-04-01 03:58:32,409 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/5edc4697f58f464f84b37fb7c52a5859 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859 2023-04-01 03:58:32,413 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859, entries=150, sequenceid=644, filesize=11.9 K 2023-04-01 03:58:32,414 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/cc7496cbd3314a70818ad881c1b1b30f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f 2023-04-01 03:58:32,417 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f, entries=150, sequenceid=644, filesize=11.9 K 2023-04-01 03:58:32,418 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/c99a540de9a244588cf26636713adc97 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97 2023-04-01 03:58:32,421 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97, entries=150, sequenceid=644, filesize=11.9 K 2023-04-01 03:58:32,422 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 64d06c34ee69fd6096a8db67850840d6 in 62ms, sequenceid=644, compaction requested=true 2023-04-01 03:58:32,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:32,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:32,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:32,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:32,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:32,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,424 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:32,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c810101[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:32,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:32,425 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,427 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:32,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:32,427 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:32,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,427 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:32,429 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:32,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:32,440 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:32,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:32,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:32,440 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:32,440 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:32,443 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,443 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,443 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:32,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:32,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,449 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:32,449 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:32,449 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:32,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:58:32,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:32,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:32,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:32,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:32,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/391dd0d5438e41c1a297554a2e347aa6 2023-04-01 03:58:32,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/46f7a7dff3694358aacafdf96b457361 2023-04-01 03:58:32,549 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:32,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:32,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:32,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:32,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:32,570 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:32,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:32,570 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:32,571 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,571 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,572 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:32,572 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,574 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,574 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:32,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321512553, max timestamp=1680321512577Num uniques 859; 2023-04-01 03:58:32,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:32,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321512553, max timestamp=1680321512577Num uniques 859; 2023-04-01 03:58:32,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680321512553, max timestamp=1680321512577Num uniques 859; 2023-04-01 03:58:32,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:32,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 03:58:32,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:32,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:32,582 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:32,582 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,582 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:32,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:32,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:32,588 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,588 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:32,588 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:32,588 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:32,588 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:32,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:32,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:32,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,594 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,594 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b2ae281[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:32,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:32,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:32,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:32,603 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:32,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:32,608 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:32,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:32,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:32,608 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:32,608 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:32,611 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,611 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,611 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,612 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,615 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:32,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:32,615 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,616 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:32,616 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:32,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:32,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321572639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321572639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321572652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,716 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:32,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:32,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:32,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:32,719 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:32,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:32,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:32,719 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,720 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,721 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,721 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:32,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:32,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:32,727 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:32,727 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:32,729 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,729 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:32,729 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:32,729 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,729 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:32,729 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:32,729 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,729 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:32,729 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:32,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:32,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:32,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5519c5bd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:32,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,736 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:32,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:32,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:32,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:32,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:32,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:32,742 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:32,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:32,742 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:32,743 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,744 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,744 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,745 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,748 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:32,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:32,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,749 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,749 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,749 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:32,749 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:32,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321572752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321572753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321572762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,849 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:32,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:32,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:32,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:32,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:32,853 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:32,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:32,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:32,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,854 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,854 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:32,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:32,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:32,856 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,856 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:32,857 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:32,857 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:32,858 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,858 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,858 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,858 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:32,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:32,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412d851c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:32,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,861 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,862 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:32,862 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:32,862 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:32,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:32,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,863 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:32,863 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,863 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:32,865 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:32,865 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:32,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:32,865 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:32,865 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:32,865 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:32,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:32,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:32,868 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,872 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,872 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:32,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,872 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:32,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:32,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321572876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321572880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/d93ae3913b0948899e6808c303293183 2023-04-01 03:58:32,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/391dd0d5438e41c1a297554a2e347aa6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6 2023-04-01 03:58:32,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321572960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6, entries=200, sequenceid=654, filesize=14.3 K 2023-04-01 03:58:32,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/46f7a7dff3694358aacafdf96b457361 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361 2023-04-01 03:58:32,972 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:32,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:32,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:32,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321572962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:32,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321572970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:32,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:32,993 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:32,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:32,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:32,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:32,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:32,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:32,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:32,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:32,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:33,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361, entries=150, sequenceid=654, filesize=11.9 K 2023-04-01 03:58:33,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/d93ae3913b0948899e6808c303293183 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183 2023-04-01 03:58:33,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183, entries=150, sequenceid=654, filesize=11.9 K 2023-04-01 03:58:33,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 64d06c34ee69fd6096a8db67850840d6 in 502ms, sequenceid=654, compaction requested=true 2023-04-01 03:58:33,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:33,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:33,005 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:58:33,005 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:58:33,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:33,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66152 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63712 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:33,007 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:33,007 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:33,007 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/253d6d20838c4246a3cba26936087ea2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=64.6 K 2023-04-01 03:58:33,007 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/e17f0baef29e4568b44b8251a778d956, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=62.2 K 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 253d6d20838c4246a3cba26936087ea2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e17f0baef29e4568b44b8251a778d956, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 70b007854d974db3bb9d74526683f619, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680321510669 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2d688dce30634d4eb0ed83d4bcfe2a3a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680321510669 2023-04-01 03:58:33,007 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4feaf550a2a648e3bda7a81d243aa838, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680321510714 2023-04-01 03:58:33,008 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 449bcbb36fd4450c938a695949ea7033, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680321510714 2023-04-01 03:58:33,008 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5edc4697f58f464f84b37fb7c52a5859, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680321511333 2023-04-01 03:58:33,008 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting cc7496cbd3314a70818ad881c1b1b30f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680321511333 2023-04-01 03:58:33,008 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 391dd0d5438e41c1a297554a2e347aa6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512485 2023-04-01 03:58:33,008 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 46f7a7dff3694358aacafdf96b457361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512497 2023-04-01 03:58:33,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:33,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:33,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:33,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:33,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:33,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:33,010 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:33,010 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:33,010 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:33,011 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:33,011 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:33,028 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#421 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:33,041 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#422 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:33,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:33,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:33,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321573300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321573301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321573302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321573410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321573410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321573411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,494 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/323bd02b06ac4ca5a60b77246cbfe989 2023-04-01 03:58:33,500 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/1c2d107021164c2ebda8a760d4863e82 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/1c2d107021164c2ebda8a760d4863e82 2023-04-01 03:58:33,500 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/703d0735245746c0a740984f1936d256 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/703d0735245746c0a740984f1936d256 2023-04-01 03:58:33,506 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into 1c2d107021164c2ebda8a760d4863e82(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:33,506 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:33,507 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=11, startTime=1680321513005; duration=0sec 2023-04-01 03:58:33,507 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:33,507 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 03:58:33,508 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 703d0735245746c0a740984f1936d256(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:33,508 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:33,508 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=11, startTime=1680321513005; duration=0sec 2023-04-01 03:58:33,508 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:33,510 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 03:58:33,510 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:33,510 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:33,510 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c236152eed164695ac7914cda396dea7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=59.8 K 2023-04-01 03:58:33,510 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c236152eed164695ac7914cda396dea7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680321510566 2023-04-01 03:58:33,511 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7f1f507b4d094db8b7f8b698b32a054f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1680321510669 2023-04-01 03:58:33,511 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 90fef2e9368c4b83a8ace126815b3037, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1680321510714 2023-04-01 03:58:33,512 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c99a540de9a244588cf26636713adc97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680321511333 2023-04-01 03:58:33,512 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d93ae3913b0948899e6808c303293183, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512497 2023-04-01 03:58:33,521 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#424 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:33,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321573618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321573619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321573620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,921 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/5f4fc4628158457bb8bf916c519855ce 2023-04-01 03:58:33,929 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/8cb064192139497f997a807721388d86 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8cb064192139497f997a807721388d86 2023-04-01 03:58:33,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321573924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321573925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321573926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:33,935 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 8cb064192139497f997a807721388d86(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:33,935 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:33,935 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=11, startTime=1680321513006; duration=0sec 2023-04-01 03:58:33,935 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:34,340 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/8fde6e8bd3b74a46ad6e94aa5865644e 2023-04-01 03:58:34,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/323bd02b06ac4ca5a60b77246cbfe989 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989 2023-04-01 03:58:34,348 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989, entries=150, sequenceid=690, filesize=11.9 K 2023-04-01 03:58:34,349 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/5f4fc4628158457bb8bf916c519855ce as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce 2023-04-01 03:58:34,364 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce, entries=150, sequenceid=690, filesize=11.9 K 2023-04-01 03:58:34,365 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/8fde6e8bd3b74a46ad6e94aa5865644e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e 2023-04-01 03:58:34,370 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e, entries=150, sequenceid=690, filesize=11.9 K 2023-04-01 03:58:34,370 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 64d06c34ee69fd6096a8db67850840d6 in 1359ms, sequenceid=690, compaction requested=false 2023-04-01 03:58:34,370 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:34,370 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,371 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:34,371 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:34,371 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,374 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:34,374 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:34,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:34,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15701d31[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:34,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:34,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,379 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:34,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:34,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,383 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:34,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:34,387 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:34,387 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:34,387 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:34,387 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:34,387 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:34,389 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,389 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,389 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:34,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:34,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,397 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:34,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,397 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:34,397 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,397 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:34,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,397 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:34,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:34,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:34,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/34087fa98b9d4df5b1eadd012e1d8481 2023-04-01 03:58:34,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321574464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321574468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321514454, max timestamp=1680321514473Num uniques 859; 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321514454, max timestamp=1680321514473Num uniques 859; 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321514454, max timestamp=1680321514473Num uniques 859; 2023-04-01 03:58:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:34,497 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:34,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:34,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:34,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:34,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:34,500 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:34,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:34,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:34,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:34,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:34,518 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,518 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:34,518 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:34,518 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:34,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:34,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4992cbf4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:34,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:34,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:34,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:34,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:34,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:34,529 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:34,532 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:34,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:34,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:34,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:34,532 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:34,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,537 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:34,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:34,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,537 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:34,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,537 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,537 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:34,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321574580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321574580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321574581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,637 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:34,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:34,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:34,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:34,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:34,640 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:34,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:34,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:34,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:34,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:34,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:34,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:34,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:34,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:34,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,644 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:34,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52bdcb56[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:34,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:34,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:34,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:34,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:34,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:34,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:34,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:34,647 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:34,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:34,649 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:34,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:34,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:34,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:34,649 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:34,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:34,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:34,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,654 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,654 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321574686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,754 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:34,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:34,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:34,757 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:34,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:34,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:34,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,758 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:34,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:34,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:34,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:34,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:34,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:34,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:34,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f15b178[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:34,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:34,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:34,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:34,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:34,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:34,769 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,769 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,769 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,769 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:34,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:34,771 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:34,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:34,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:34,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:34,771 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:34,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321574782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321574783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:34,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,800 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:34,800 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:34,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:34,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/617fb97f31e5400d9677af0a0370d848 2023-04-01 03:58:34,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321574893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,901 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:34,901 DEBUG [Thread-2913] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:34,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:34,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:34,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:34,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:34,908 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:34,908 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:34,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:34,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,909 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321574897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/74d71607aa9e4b7b85135c8aaac07caf 2023-04-01 03:58:34,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:34,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321574902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,915 DEBUG [Thread-2915] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:34,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/34087fa98b9d4df5b1eadd012e1d8481 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481 2023-04-01 03:58:34,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481, entries=200, sequenceid=704, filesize=14.3 K 2023-04-01 03:58:34,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/617fb97f31e5400d9677af0a0370d848 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848 2023-04-01 03:58:34,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848, entries=150, sequenceid=704, filesize=11.9 K 2023-04-01 03:58:34,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/74d71607aa9e4b7b85135c8aaac07caf as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf 2023-04-01 03:58:34,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:34,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:34,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:34,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:34,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:34,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:34,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:34,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:34,946 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:34,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,962 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:34,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:34,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:34,963 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:34,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf, entries=150, sequenceid=704, filesize=11.9 K 2023-04-01 03:58:34,966 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-01 03:58:34,966 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-01 03:58:34,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 64d06c34ee69fd6096a8db67850840d6 in 532ms, sequenceid=704, compaction requested=true 2023-04-01 03:58:34,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:34,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:34,968 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:34,968 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:34,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:34,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:34,969 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,969 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:34,969 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/1c2d107021164c2ebda8a760d4863e82, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=38.6 K 2023-04-01 03:58:34,969 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/703d0735245746c0a740984f1936d256, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.2 K 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 1c2d107021164c2ebda8a760d4863e82, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512497 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 703d0735245746c0a740984f1936d256, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512497 2023-04-01 03:58:34,969 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 323bd02b06ac4ca5a60b77246cbfe989, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1680321512621 2023-04-01 03:58:34,970 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 5f4fc4628158457bb8bf916c519855ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1680321512621 2023-04-01 03:58:34,970 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 34087fa98b9d4df5b1eadd012e1d8481, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:34,970 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 617fb97f31e5400d9677af0a0370d848, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:34,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:34,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a19b23e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:34,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:34,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:34,991 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:34,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:34,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:34,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:34,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:34,993 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:34,993 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,993 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,993 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:34,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:34,996 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:34,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:34,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:34,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:34,996 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:34,997 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:34,997 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,998 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:34,998 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,000 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,000 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,001 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,001 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:35,007 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:35,047 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/35f2b2e6e8af4ea0917a6f3157a55439 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/35f2b2e6e8af4ea0917a6f3157a55439 2023-04-01 03:58:35,050 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/6bec20ff12ac4f6c9ed46e7a700855c9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6bec20ff12ac4f6c9ed46e7a700855c9 2023-04-01 03:58:35,055 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 6bec20ff12ac4f6c9ed46e7a700855c9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:35,055 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:35,055 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321514968; duration=0sec 2023-04-01 03:58:35,055 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:35,055 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:35,056 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into 35f2b2e6e8af4ea0917a6f3157a55439(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:35,056 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:35,056 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321514968; duration=0sec 2023-04-01 03:58:35,056 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:35,057 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:35,057 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:35,057 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,057 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8cb064192139497f997a807721388d86, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.2 K 2023-04-01 03:58:35,058 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8cb064192139497f997a807721388d86, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680321512497 2023-04-01 03:58:35,058 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8fde6e8bd3b74a46ad6e94aa5865644e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1680321512621 2023-04-01 03:58:35,058 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 74d71607aa9e4b7b85135c8aaac07caf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:35,066 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:35,092 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/e1f1caa773c743488e022923456b49ff as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e1f1caa773c743488e022923456b49ff 2023-04-01 03:58:35,096 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into e1f1caa773c743488e022923456b49ff(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:35,096 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-01 03:58:35,096 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:35,096 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321514968; duration=0sec 2023-04-01 03:58:35,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:35,096 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:35,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:35,101 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,106 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,108 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,118 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,118 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,133 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,133 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,135 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,135 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321575135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321575135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@358b09e2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,141 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/9e04c93cd7844179a9671dbe69f473a6 2023-04-01 03:58:35,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,147 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,147 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,151 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,154 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,175 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,175 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,176 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/de16ae39d46c49da8d57f589edd81b8a 2023-04-01 03:58:35,208 WARN [DataStreamer for file /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/9f5a9a7e276e47ff8b4183c3d0d679f6] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 03:58:35,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/9f5a9a7e276e47ff8b4183c3d0d679f6 2023-04-01 03:58:35,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/9e04c93cd7844179a9671dbe69f473a6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6 2023-04-01 03:58:35,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6, entries=200, sequenceid=744, filesize=14.3 K 2023-04-01 03:58:35,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/de16ae39d46c49da8d57f589edd81b8a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a 2023-04-01 03:58:35,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321575214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a, entries=150, sequenceid=744, filesize=11.9 K 2023-04-01 03:58:35,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/9f5a9a7e276e47ff8b4183c3d0d679f6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6 2023-04-01 03:58:35,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6, entries=150, sequenceid=744, filesize=11.9 K 2023-04-01 03:58:35,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 64d06c34ee69fd6096a8db67850840d6 in 142ms, sequenceid=744, compaction requested=false 2023-04-01 03:58:35,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:35,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:35,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 03:58:35,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:35,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:35,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:35,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:35,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:35,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:35,275 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,294 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,294 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,294 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,294 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,295 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,296 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,296 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,297 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,297 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,307 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,309 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,309 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,309 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,309 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,317 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@950837e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,319 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,320 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,320 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,320 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,320 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,321 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,324 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,324 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,324 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,324 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,324 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,326 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,328 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,330 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,330 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,331 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,331 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,331 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,331 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321575362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321575362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,431 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,438 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,438 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,442 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,444 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,453 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1c7d5d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,462 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,462 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,466 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,466 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,480 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,480 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,481 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321575474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321575474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,581 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,598 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,631 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,632 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,632 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,632 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,632 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,632 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,632 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,632 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,632 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,633 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,634 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,634 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb3bc42[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,638 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,640 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,640 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,640 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,640 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,643 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,643 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,647 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,647 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,647 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,655 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,655 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:35,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/fc88d049caba46e0bdcbdeadb0e54d44 2023-04-01 03:58:35,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321575683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321575684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321575725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,755 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,774 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,776 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,776 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,777 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,777 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,779 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,779 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,779 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,788 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119a6e2f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,790 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,790 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,792 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,792 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,793 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,793 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,796 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,796 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,797 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,797 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,798 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,800 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,801 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,801 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,901 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:35,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:35,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:35,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:35,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:35,904 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:35,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:35,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:35,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,904 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:35,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:35,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:35,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:35,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:35,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:35,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:35,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:35,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:35,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cdc4411[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:35,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:35,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,911 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:35,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,911 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:35,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:35,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:35,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:35,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,913 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:35,913 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:35,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:35,917 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:35,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:35,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:35,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:35,917 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:35,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:35,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:35,934 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,935 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:35,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,942 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,942 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:35,942 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:35,942 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:35,943 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:35,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:35,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321575992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:35,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:35,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321575993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,042 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,045 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,045 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,047 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:36,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,048 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,048 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,049 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,049 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:36,051 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24923442[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:36,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:36,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:36,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:36,054 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:36,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,054 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:36,055 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:36,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:36,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:36,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:36,056 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:36,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,057 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,059 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:36,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:36,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,060 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:36,060 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/fca9db45b6054bfaba7818fc9ebd9092 2023-04-01 03:58:36,160 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,163 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,163 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,167 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,168 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,168 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,168 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,168 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,169 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,169 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,169 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,169 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:36,169 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:36,170 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,170 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4522cd3e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:36,171 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:36,171 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,171 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,172 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:36,172 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:36,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,172 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:36,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:36,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:36,175 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:36,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:36,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:36,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:36,175 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:36,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,175 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:36,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:36,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:36,178 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:36,178 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:36,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,278 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,281 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,281 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,282 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:36,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,283 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,284 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,284 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,284 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,284 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:36,284 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:36,289 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,289 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:36,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b28a3[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:36,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:36,290 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,290 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,291 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,291 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,291 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:36,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:36,293 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:36,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:36,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:36,293 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:36,293 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:36,294 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,294 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,294 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,294 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,296 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:36,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:36,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:36,298 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,298 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:36,298 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:36,398 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,402 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,402 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,404 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:36,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,405 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,406 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,406 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,406 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,406 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:36,406 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:36,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,407 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7909088f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:36,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:36,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,409 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,409 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,409 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:36,409 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:36,411 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:36,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:36,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:36,411 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:36,411 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:36,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,412 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,413 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:36,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,416 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:36,416 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:36,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321576497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321576501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,516 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,519 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,519 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,520 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:36,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:36,521 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:36,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c8c03c8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:36,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:36,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:36,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:36,529 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:36,531 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:36,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:36,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:36,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:36,531 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:36,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/f4584d8180954ef3b6c93e802240ef4a 2023-04-01 03:58:36,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:36,533 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:36,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/fc88d049caba46e0bdcbdeadb0e54d44 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44 2023-04-01 03:58:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44, entries=200, sequenceid=756, filesize=14.3 K 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:36,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:36,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,539 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:36,539 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:36,539 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:36,539 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:36,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:36,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/fca9db45b6054bfaba7818fc9ebd9092 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092 2023-04-01 03:58:36,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:36,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092, entries=150, sequenceid=756, filesize=11.9 K 2023-04-01 03:58:36,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/f4584d8180954ef3b6c93e802240ef4a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a 2023-04-01 03:58:36,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a, entries=150, sequenceid=756, filesize=11.9 K 2023-04-01 03:58:36,550 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 64d06c34ee69fd6096a8db67850840d6 in 1306ms, sequenceid=756, compaction requested=true 2023-04-01 03:58:36,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:36,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:36,551 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:36,551 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:36,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:36,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:36,562 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:36,563 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:36,563 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,563 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6bec20ff12ac4f6c9ed46e7a700855c9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.3 K 2023-04-01 03:58:36,562 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42052 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:36,563 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:36,563 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,563 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/35f2b2e6e8af4ea0917a6f3157a55439, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=41.1 K 2023-04-01 03:58:36,564 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6bec20ff12ac4f6c9ed46e7a700855c9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:36,564 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 35f2b2e6e8af4ea0917a6f3157a55439, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:36,564 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting de16ae39d46c49da8d57f589edd81b8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680321514559 2023-04-01 03:58:36,564 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9e04c93cd7844179a9671dbe69f473a6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680321514559 2023-04-01 03:58:36,565 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fca9db45b6054bfaba7818fc9ebd9092, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:36,565 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting fc88d049caba46e0bdcbdeadb0e54d44, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:36,572 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:36,591 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:36,609 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/d15df2f3afbc45ef80b620cc7ba8a7cd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d15df2f3afbc45ef80b620cc7ba8a7cd 2023-04-01 03:58:36,613 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into d15df2f3afbc45ef80b620cc7ba8a7cd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:36,613 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:36,613 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321516551; duration=0sec 2023-04-01 03:58:36,613 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:36,613 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:36,614 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:36,614 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:36,614 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,614 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e1f1caa773c743488e022923456b49ff, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.3 K 2023-04-01 03:58:36,614 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e1f1caa773c743488e022923456b49ff, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1680321513295 2023-04-01 03:58:36,615 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9f5a9a7e276e47ff8b4183c3d0d679f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1680321514559 2023-04-01 03:58:36,615 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f4584d8180954ef3b6c93e802240ef4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:36,618 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#440 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:36,639 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:36,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:36,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:36,641 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:36,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:36,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:36,641 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,643 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:36,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:36,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,643 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:36,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:36,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:36,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:36,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:36,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:36,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:36,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:36,644 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:36,644 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:36,644 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:58:36,644 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:36,645 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:36,645 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:36,645 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:36,645 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:36,645 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:36,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:36,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:36,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:36,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321576787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:36,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:36,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321576895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,007 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/df0e1a06a6f542099b7cbaa9d4e17d91 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/df0e1a06a6f542099b7cbaa9d4e17d91 2023-04-01 03:58:37,011 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into df0e1a06a6f542099b7cbaa9d4e17d91(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:37,011 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:37,011 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321516551; duration=0sec 2023-04-01 03:58:37,011 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:37,026 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/2e0ee72066564777947a54c1dd951d44 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2e0ee72066564777947a54c1dd951d44 2023-04-01 03:58:37,031 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 2e0ee72066564777947a54c1dd951d44(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:37,031 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:37,031 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321516551; duration=0sec 2023-04-01 03:58:37,031 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:37,056 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/b075494102f94e56a53f8a89ddd9982a 2023-04-01 03:58:37,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:37,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321577101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:37,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321577405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,470 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/2af4df768acd47a0860ee49d10ea26e5 2023-04-01 03:58:37,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:37,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321577500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,508 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=777 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/3b7c88d98426409ea4a846285d743779 2023-04-01 03:58:37,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:37,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321577507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,517 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/b075494102f94e56a53f8a89ddd9982a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a 2023-04-01 03:58:37,520 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a, entries=150, sequenceid=777, filesize=11.9 K 2023-04-01 03:58:37,521 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/2af4df768acd47a0860ee49d10ea26e5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5 2023-04-01 03:58:37,525 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5, entries=150, sequenceid=777, filesize=11.9 K 2023-04-01 03:58:37,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/3b7c88d98426409ea4a846285d743779 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779 2023-04-01 03:58:37,529 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779, entries=150, sequenceid=777, filesize=11.9 K 2023-04-01 03:58:37,530 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 64d06c34ee69fd6096a8db67850840d6 in 886ms, sequenceid=777, compaction requested=false 2023-04-01 03:58:37,530 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:37,530 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:37,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:37,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:37,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:37,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:37,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:37,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:37,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f147c5a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:37,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:37,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,537 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,537 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:37,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:37,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:37,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:37,543 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:37,544 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:37,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:37,546 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:37,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:37,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:37,546 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:37,546 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:37,546 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:37,547 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:37,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:37,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:37,549 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:37,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:37,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:37,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:37,550 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:37,550 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:37,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:37,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,650 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:37,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:37,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:37,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:37,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:37,653 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:37,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:37,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:37,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:37,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:37,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,661 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:37,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:37,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:37,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:37,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:37,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:37,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:37,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:37,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:37,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:37,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:37,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:37,666 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:37,666 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:37,667 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:37,691 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/b1933ab2f16c4a4586c9bafab667c1eb 2023-04-01 03:58:37,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:37,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:37,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:37,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321577992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321578098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,114 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/0ee5061e8adc4c28ab5f797f9bab73b6 2023-04-01 03:58:38,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321578306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,532 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/54607a65e62e4201a9af0760aae67b50 2023-04-01 03:58:38,534 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/b1933ab2f16c4a4586c9bafab667c1eb as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb 2023-04-01 03:58:38,539 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb, entries=150, sequenceid=791, filesize=11.9 K 2023-04-01 03:58:38,540 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/0ee5061e8adc4c28ab5f797f9bab73b6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6 2023-04-01 03:58:38,545 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6, entries=150, sequenceid=791, filesize=11.9 K 2023-04-01 03:58:38,546 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/54607a65e62e4201a9af0760aae67b50 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50 2023-04-01 03:58:38,550 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50, entries=150, sequenceid=791, filesize=11.9 K 2023-04-01 03:58:38,551 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 64d06c34ee69fd6096a8db67850840d6 in 885ms, sequenceid=791, compaction requested=true 2023-04-01 03:58:38,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:38,551 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:38,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:38,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:38,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:38,553 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,553 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,553 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:38,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:38,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4104912b[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:38,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:38,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,598 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:38,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:38,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:38,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,599 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:38,599 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,600 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:38,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:38,601 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:38,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:38,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:38,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:38,601 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:38,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,603 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,614 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680321517960, max timestamp=1680321518613Num uniques 859; 2023-04-01 03:58:38,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 03:58:38,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:38,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:38,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:38,624 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:38,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,625 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,625 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:38,625 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:38,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321578700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,725 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:38,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:38,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:38,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:38,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:38,728 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:38,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:38,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:38,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,730 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,730 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,730 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,730 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:38,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:38,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:38,732 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,732 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:38,732 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:38,732 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:38,734 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f8a4f4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:38,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:38,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,735 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:38,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,736 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:38,736 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,736 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:38,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:38,738 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:38,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:38,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:38,738 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:38,738 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:38,739 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,739 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,740 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:38,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,742 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:38,742 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:38,742 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321578809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,842 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:38,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:38,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:38,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:38,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:38,845 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:38,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:38,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:38,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,851 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,851 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,851 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,851 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:38,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:38,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:38,852 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:38,852 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:38,852 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:38,852 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:38,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:38,854 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,854 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:38,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c466b60[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:38,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:38,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,856 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,856 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,856 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:38,856 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:38,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:38,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:38,860 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:38,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:38,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:38,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:38,860 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:38,869 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,869 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,869 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,871 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:38,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,873 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:38,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,874 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:38,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50506 deadline: 1680321578909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,915 DEBUG [Thread-2913] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:38,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50512 deadline: 1680321578955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,959 DEBUG [Thread-2915] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8231 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:38,974 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:38,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:38,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:38,976 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:38,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:38,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:38,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,977 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:38,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:38,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:38,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:38,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:38,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:38,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:38,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:38,980 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,980 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,980 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,981 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:38,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:38,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608c2a1e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:38,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:38,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,983 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:38,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:38,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:38,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:38,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:38,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:38,986 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:38,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:38,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:38,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:38,986 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:38,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:38,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:38,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,989 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:38,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:38,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:38,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:38,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:38,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:38,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:38,991 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:38,991 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:38,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321579014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/c33e167c3db04a3b8c14c0290275d229 2023-04-01 03:58:39,091 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,094 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,094 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,105 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,105 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,105 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,106 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,106 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,107 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,107 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,107 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,107 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56472ed[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,110 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,110 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,112 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,112 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,114 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,114 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,114 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,115 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,115 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,115 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,117 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,119 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,119 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,119 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,219 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,222 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,224 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,225 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,226 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,226 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,226 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,226 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,228 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,228 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@308eb9b1[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,233 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,235 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,235 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,237 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,238 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,244 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,244 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321579322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,344 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,346 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,352 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,352 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,352 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,353 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7cd9ab[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,355 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,355 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,356 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,356 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,356 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,358 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,358 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,359 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,362 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,363 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,363 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/6f34289ed13f434db362202bbd109bdf 2023-04-01 03:58:39,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/cd1cd282790945d1b80d345ac57d7bd0 2023-04-01 03:58:39,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/c33e167c3db04a3b8c14c0290275d229 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229 2023-04-01 03:58:39,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229, entries=200, sequenceid=812, filesize=14.3 K 2023-04-01 03:58:39,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/6f34289ed13f434db362202bbd109bdf as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf 2023-04-01 03:58:39,463 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,466 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf, entries=150, sequenceid=812, filesize=11.9 K 2023-04-01 03:58:39,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/cd1cd282790945d1b80d345ac57d7bd0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0 2023-04-01 03:58:39,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,467 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0, entries=150, sequenceid=812, filesize=11.9 K 2023-04-01 03:58:39,469 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,469 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,469 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,470 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=87.22 KB/89310 for 64d06c34ee69fd6096a8db67850840d6 in 855ms, sequenceid=812, compaction requested=true 2023-04-01 03:58:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:39,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:39,470 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:39,470 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:39,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:39,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:39,471 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,471 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:39,471 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:39,471 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:39,471 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,471 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:39,471 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/df0e1a06a6f542099b7cbaa9d4e17d91, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=48.3 K 2023-04-01 03:58:39,471 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,472 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d15df2f3afbc45ef80b620cc7ba8a7cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=50.7 K 2023-04-01 03:58:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,472 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting d15df2f3afbc45ef80b620cc7ba8a7cd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:39,472 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting df0e1a06a6f542099b7cbaa9d4e17d91, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,472 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b075494102f94e56a53f8a89ddd9982a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1680321515338 2023-04-01 03:58:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,472 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b1933ab2f16c4a4586c9bafab667c1eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1680321516767 2023-04-01 03:58:39,472 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2af4df768acd47a0860ee49d10ea26e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1680321515338 2023-04-01 03:58:39,473 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c33e167c3db04a3b8c14c0290275d229, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,473 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0ee5061e8adc4c28ab5f797f9bab73b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1680321516767 2023-04-01 03:58:39,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,473 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6f34289ed13f434db362202bbd109bdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:39,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e21d480[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,474 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,477 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,477 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,478 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#450 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:39,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,479 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,479 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,482 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#451 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,482 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,483 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,483 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,483 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:39,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:39,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:39,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321579575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321579580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,583 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,586 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,586 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,587 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,587 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,589 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,591 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787426b3[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,592 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,593 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,593 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,593 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,596 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,596 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,597 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,598 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,602 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,602 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321579681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321579682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,703 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,705 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,705 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,706 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,706 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,706 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,706 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,707 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,710 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,711 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,716 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,717 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3df25b3d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,725 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,725 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,725 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,725 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,725 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,735 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,735 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,735 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,741 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,742 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,750 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,750 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,750 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,750 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321579828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,850 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,856 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,856 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,856 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,856 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,857 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,859 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,859 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,861 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,861 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,861 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,861 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,861 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,863 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,863 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,864 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,864 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa7ca9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,866 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,866 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,867 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,869 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,869 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,869 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,870 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,870 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,871 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:39,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,872 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,873 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,873 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,888 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/29e5689510d44cd2bbc926707473191b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/29e5689510d44cd2bbc926707473191b 2023-04-01 03:58:39,890 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/2f26bb2039604c2cb18890ce4e5a45d2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2f26bb2039604c2cb18890ce4e5a45d2 2023-04-01 03:58:39,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321579886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:39,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321579887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,893 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into 29e5689510d44cd2bbc926707473191b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:39,893 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:39,893 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=12, startTime=1680321519470; duration=0sec 2023-04-01 03:58:39,893 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:39,893 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:39,893 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 2f26bb2039604c2cb18890ce4e5a45d2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:39,893 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:39,893 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=12, startTime=1680321519470; duration=0sec 2023-04-01 03:58:39,894 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:39,894 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:39,894 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:39,894 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,894 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2e0ee72066564777947a54c1dd951d44, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=48.3 K 2023-04-01 03:58:39,894 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2e0ee72066564777947a54c1dd951d44, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680321515124 2023-04-01 03:58:39,894 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 3b7c88d98426409ea4a846285d743779, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=777, earliestPutTs=1680321515338 2023-04-01 03:58:39,895 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 54607a65e62e4201a9af0760aae67b50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1680321516767 2023-04-01 03:58:39,895 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting cd1cd282790945d1b80d345ac57d7bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:39,901 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:39,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/6c0dadaf2e024bd1affb04518172a454 2023-04-01 03:58:39,973 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:39,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:39,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:39,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:39,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:39,976 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:39,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:39,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:39,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,976 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:39,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:39,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:39,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:39,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:39,979 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:39,979 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:39,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,981 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:39,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54af2d62[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:39,982 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:39,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,983 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:39,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:39,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:39,983 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:39,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:39,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:39,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:39,986 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:39,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:39,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:39,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:39,987 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:39,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:39,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:39,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,989 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:39,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:39,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:39,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:39,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:39,990 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:39,990 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:39,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,090 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,093 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,095 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,096 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,096 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,096 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,096 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:40,097 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:40,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d37c8fa[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:40,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:40,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:40,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:40,101 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:40,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:40,103 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:40,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:40,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:40,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:40,103 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:40,105 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,105 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,105 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,105 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,106 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:40,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,109 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,109 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:40,109 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:40,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321580192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:40,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321580193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,209 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,233 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,243 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,254 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,254 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,254 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,254 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,254 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:40,255 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:40,256 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:40,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@223351d2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:40,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:40,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,260 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:40,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:40,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:40,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:40,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:40,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:40,265 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:40,265 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:40,265 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:40,265 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:40,265 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:40,266 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,266 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,266 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,266 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,269 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:40,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:40,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,271 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,271 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:40,271 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:40,271 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,271 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,271 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:40,309 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/278794eaecd245d9805b8a725eb6c614 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/278794eaecd245d9805b8a725eb6c614 2023-04-01 03:58:40,313 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 278794eaecd245d9805b8a725eb6c614(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:40,313 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:40,313 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=12, startTime=1680321519470; duration=0sec 2023-04-01 03:58:40,313 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:40,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/bd7de73e7bc24546b7f37871f1702db4 2023-04-01 03:58:40,371 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,373 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,374 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,374 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,374 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,374 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,374 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,376 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,377 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,377 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,377 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,377 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:40,377 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:40,378 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:40,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75fb7e09[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:40,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:40,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,380 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,380 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:40,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:40,381 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:40,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:40,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:40,383 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:40,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:40,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:40,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:40,383 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:40,384 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,384 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,384 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,407 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:40,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:40,408 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:40,408 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,408 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:40,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,508 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,514 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,514 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,516 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,522 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,522 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,523 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,523 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:40,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:40,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:40,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:40,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@589299e9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:40,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:40,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,531 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:40,531 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:40,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,531 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:40,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:40,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:40,533 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:40,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:40,535 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:40,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:40,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:40,535 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:40,535 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:40,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:40,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:40,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,542 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:40,542 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:40,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:40,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,642 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,647 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,647 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,662 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:40,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:40,664 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:40,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b51fa19[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:40,665 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:40,665 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,666 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:40,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:40,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:40,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:40,667 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:40,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:40,669 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:40,669 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:40,669 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:40,670 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:40,670 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,670 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:40,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,671 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:40,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:40,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:40,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:40,673 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,674 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:40,674 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:40,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:40,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:40,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321580697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321580704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/fad40ee75b404aa0bb2cbfb21f020460 2023-04-01 03:58:40,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/6c0dadaf2e024bd1affb04518172a454 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454 2023-04-01 03:58:40,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454, entries=200, sequenceid=829, filesize=14.3 K 2023-04-01 03:58:40,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/bd7de73e7bc24546b7f37871f1702db4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4 2023-04-01 03:58:40,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4, entries=150, sequenceid=829, filesize=11.9 K 2023-04-01 03:58:40,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/fad40ee75b404aa0bb2cbfb21f020460 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460 2023-04-01 03:58:40,774 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:40,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:40,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460, entries=150, sequenceid=829, filesize=11.9 K 2023-04-01 03:58:40,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:40,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 64d06c34ee69fd6096a8db67850840d6 in 1248ms, sequenceid=829, compaction requested=false 2023-04-01 03:58:40,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:40,778 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:40,778 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,778 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:40,778 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:40,778 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,778 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,779 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:40,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:40,780 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:40,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:40,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:40,780 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:40,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:40,780 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:40,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:40,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:40,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:40,780 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:40,781 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:40,781 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:40,781 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:40,781 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:40,789 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=847 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/ebe1769e8693456a92e8e0ca36024b2a 2023-04-01 03:58:40,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:40,855 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:40,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:40,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321580950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321581053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,198 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=847 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/59bd385adc0d4edf9533d93c2b65c80c 2023-04-01 03:58:41,208 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=847 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/0af13bde649f439bbb182d9003bedad2 2023-04-01 03:58:41,211 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/ebe1769e8693456a92e8e0ca36024b2a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a 2023-04-01 03:58:41,213 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a, entries=150, sequenceid=847, filesize=11.9 K 2023-04-01 03:58:41,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/59bd385adc0d4edf9533d93c2b65c80c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c 2023-04-01 03:58:41,216 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c, entries=150, sequenceid=847, filesize=11.9 K 2023-04-01 03:58:41,217 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/0af13bde649f439bbb182d9003bedad2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2 2023-04-01 03:58:41,220 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2, entries=150, sequenceid=847, filesize=11.9 K 2023-04-01 03:58:41,221 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 64d06c34ee69fd6096a8db67850840d6 in 440ms, sequenceid=847, compaction requested=true 2023-04-01 03:58:41,221 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:41,221 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:41,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:41,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:41,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:41,222 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,222 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,223 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:41,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53549b60[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:41,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:41,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,226 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:41,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:41,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,226 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:41,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:41,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,227 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:41,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:41,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:41,229 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:41,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:41,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:41,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:41,230 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:41,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:41,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,235 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,235 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:41,235 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:41,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:41,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:41,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:41,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,335 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:41,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:41,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:41,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:41,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:41,339 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:41,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:41,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:41,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,339 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:41,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:41,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:41,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:41,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:41,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:41,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:41,343 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,343 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:41,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:41,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:41,345 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:41,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d71ac35[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:41,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:41,346 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,348 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:41,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:41,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:41,352 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:41,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:41,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:41,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:41,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:41,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:41,355 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:41,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:41,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:41,359 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,360 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,360 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,360 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,360 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:41,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321581353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,360 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:41,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,360 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,460 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:41,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:41,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:41,465 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:41,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:41,465 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:41,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,466 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,467 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:41,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321581462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:41,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:41,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:41,477 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:41,477 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,478 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:41,478 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:41,478 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:41,490 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,492 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,492 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e02fa3[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:41,494 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:41,494 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,494 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,494 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:41,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:41,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:41,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:41,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:41,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,495 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:41,495 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:41,497 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:41,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:41,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:41,498 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:41,498 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:41,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,498 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,499 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,499 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,500 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,501 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,501 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,502 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:41,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,502 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:41,502 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:41,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,602 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:41,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:41,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:41,606 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:41,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:41,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:41,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,606 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:41,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:41,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:41,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:41,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:41,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:41,612 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55058ab4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:41,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:41,613 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,616 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:41,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:41,616 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,616 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:41,616 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:41,617 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:41,617 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,617 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,618 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:41,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:41,626 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:41,626 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:41,626 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:41,626 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:41,626 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:41,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:41,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,633 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:41,634 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:41,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:41,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:41,642 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:41,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:41,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:41,642 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:41,643 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:41,643 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:41,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:41,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/4b27f6c0dbe346a7a3e1030ee017a3d6 2023-04-01 03:58:41,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321581673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/53a0671a01c340cf8cab274dfc1b09f3 2023-04-01 03:58:41,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321581704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:41,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321581711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=865 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/117ef7a81c5c4d2b9f4d7ecd1011c972 2023-04-01 03:58:41,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/4b27f6c0dbe346a7a3e1030ee017a3d6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6 2023-04-01 03:58:41,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6, entries=200, sequenceid=865, filesize=14.3 K 2023-04-01 03:58:41,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/53a0671a01c340cf8cab274dfc1b09f3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3 2023-04-01 03:58:41,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3, entries=150, sequenceid=865, filesize=11.9 K 2023-04-01 03:58:41,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/117ef7a81c5c4d2b9f4d7ecd1011c972 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972 2023-04-01 03:58:41,743 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:41,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972, entries=150, sequenceid=865, filesize=11.9 K 2023-04-01 03:58:41,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:41,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 64d06c34ee69fd6096a8db67850840d6 in 483ms, sequenceid=865, compaction requested=true 2023-04-01 03:58:41,745 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:41,745 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:41,745 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:41,745 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:41,745 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:41,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:41,746 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:41,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,745 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:41,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:41,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:41,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,746 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,746 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54476 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:41,747 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:41,747 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,747 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/29e5689510d44cd2bbc926707473191b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=53.2 K 2023-04-01 03:58:41,747 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:41,747 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:41,747 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,747 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2f26bb2039604c2cb18890ce4e5a45d2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=48.4 K 2023-04-01 03:58:41,748 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 29e5689510d44cd2bbc926707473191b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:41,748 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2f26bb2039604c2cb18890ce4e5a45d2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:41,748 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6c0dadaf2e024bd1affb04518172a454, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680321518622 2023-04-01 03:58:41,748 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting ebe1769e8693456a92e8e0ca36024b2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=847, earliestPutTs=1680321519568 2023-04-01 03:58:41,748 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4b27f6c0dbe346a7a3e1030ee017a3d6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520907 2023-04-01 03:58:41,749 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bd7de73e7bc24546b7f37871f1702db4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680321518622 2023-04-01 03:58:41,749 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 59bd385adc0d4edf9533d93c2b65c80c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=847, earliestPutTs=1680321519568 2023-04-01 03:58:41,750 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 53a0671a01c340cf8cab274dfc1b09f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520932 2023-04-01 03:58:41,755 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:41,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:41,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:41,756 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#462 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:41,756 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:41,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:41,757 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:41,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:41,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:41,770 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:41,770 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:41,771 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:41,771 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:41,771 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:41,772 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:41,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:41,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:42,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:42,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321582062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321582166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,175 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/b1cbd4e6548b4fd78151e271f5b2fed5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b1cbd4e6548b4fd78151e271f5b2fed5 2023-04-01 03:58:42,177 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into b1cbd4e6548b4fd78151e271f5b2fed5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:42,177 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:42,177 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=12, startTime=1680321521745; duration=0sec 2023-04-01 03:58:42,178 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:42,178 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:58:42,178 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:58:42,178 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:42,178 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,178 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/278794eaecd245d9805b8a725eb6c614, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=48.4 K 2023-04-01 03:58:42,179 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 278794eaecd245d9805b8a725eb6c614, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1680321517960 2023-04-01 03:58:42,179 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fad40ee75b404aa0bb2cbfb21f020460, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680321518622 2023-04-01 03:58:42,179 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0af13bde649f439bbb182d9003bedad2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=847, earliestPutTs=1680321519568 2023-04-01 03:58:42,180 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 117ef7a81c5c4d2b9f4d7ecd1011c972, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520932 2023-04-01 03:58:42,182 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/f4bad75ecbd942e698a09e58b489d617 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f4bad75ecbd942e698a09e58b489d617 2023-04-01 03:58:42,184 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:42,185 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into f4bad75ecbd942e698a09e58b489d617(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:42,185 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:42,185 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=12, startTime=1680321521745; duration=0sec 2023-04-01 03:58:42,185 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:42,187 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/862991d2c0794d1c9ba161e9254d6567 2023-04-01 03:58:42,194 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/050b849b312e4729af702eaf188522c1 2023-04-01 03:58:42,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:42,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321582372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,593 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/46e98d70eb52482f96f486a2ad22a37d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/46e98d70eb52482f96f486a2ad22a37d 2023-04-01 03:58:42,597 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 46e98d70eb52482f96f486a2ad22a37d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:42,597 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:42,597 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=12, startTime=1680321521745; duration=0sec 2023-04-01 03:58:42,597 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:42,601 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/ce981187733642aba942618d70e9b927 2023-04-01 03:58:42,605 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/862991d2c0794d1c9ba161e9254d6567 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567 2023-04-01 03:58:42,609 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567, entries=150, sequenceid=879, filesize=11.9 K 2023-04-01 03:58:42,610 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/050b849b312e4729af702eaf188522c1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1 2023-04-01 03:58:42,613 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1, entries=150, sequenceid=879, filesize=11.9 K 2023-04-01 03:58:42,613 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/ce981187733642aba942618d70e9b927 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927 2023-04-01 03:58:42,621 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927, entries=150, sequenceid=879, filesize=11.9 K 2023-04-01 03:58:42,622 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 64d06c34ee69fd6096a8db67850840d6 in 851ms, sequenceid=879, compaction requested=false 2023-04-01 03:58:42,622 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:42,622 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:42,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:42,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:42,622 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:42,625 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,625 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,626 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:42,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227d84ef[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:42,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:42,627 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,628 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,628 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:42,628 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:42,628 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:42,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,629 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:42,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:42,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,629 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:42,630 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:42,632 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:42,632 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:42,632 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:42,632 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:42,632 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:42,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,650 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:42,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,660 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:42,660 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:42,660 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:42,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:42,660 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:42,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:42,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:42,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/6d1ab37e696e4eafac6b82c0b4c66582 2023-04-01 03:58:42,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:42,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321582739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,760 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:42,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:42,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:42,770 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:42,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:42,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:42,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:42,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,773 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:42,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:42,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,773 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:42,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:42,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:42,774 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:42,774 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:42,774 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:42,782 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,782 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:42,782 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:42,782 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:42,791 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,791 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,792 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:42,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@529d9491[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:42,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:42,794 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,808 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,808 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:42,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:42,808 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:42,808 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:42,809 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:42,809 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,809 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,809 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:42,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:42,811 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:42,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:42,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:42,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:42,812 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:42,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,813 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,814 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,815 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:42,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:42,815 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:42,816 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:42,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,816 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:42,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:42,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321582843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,916 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:42,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:42,918 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:42,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:42,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:42,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:42,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,920 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:42,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:42,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:42,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:42,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:42,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:42,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:42,922 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:42,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,923 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:42,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b02456[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:42,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:42,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,927 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:42,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:42,927 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:42,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,928 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:42,929 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:42,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:42,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:42,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:42,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:42,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:42,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:42,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:42,934 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,934 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:42,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,934 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:42,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:42,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:42,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:42,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:42,938 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:42,938 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:42,938 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:42,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,038 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:43,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:43,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:43,044 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:43,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:43,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:43,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,044 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,046 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,046 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:43,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:43,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:43,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:43,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:43,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:43,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:43,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:43,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:43,048 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,048 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:43,049 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:43,049 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:43,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:43,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:43,049 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321583047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,059 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,059 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:43,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b19529e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:43,061 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:43,061 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,062 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:43,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:43,062 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,062 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:43,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:43,063 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:43,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,063 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:43,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:43,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:43,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:43,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:43,066 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:43,066 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:43,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,067 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,067 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,069 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,071 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,071 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:43,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:43,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:43,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,072 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:43,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,072 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:43,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/8f42be692e304bf59e8bec128d2eb010 2023-04-01 03:58:43,172 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:43,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:43,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:43,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:43,175 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:43,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:43,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:43,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,191 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,191 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:43,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:43,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:43,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:43,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:43,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:43,192 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:43,192 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,192 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:43,192 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:43,192 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:43,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:43,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:43,193 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:43,200 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,200 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@558de7ee[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:43,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:43,202 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,216 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:43,217 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,217 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:43,218 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:43,219 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:43,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:43,219 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:43,220 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:43,220 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:43,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,221 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,222 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,223 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:43,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,223 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,223 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:43,224 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:43,324 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:43,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:43,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:43,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:43,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:43,326 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:43,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:43,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:43,327 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,327 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,329 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,329 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:43,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:43,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:43,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:43,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:43,330 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:43,332 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,332 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b8ac8d2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:43,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:43,333 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,334 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:43,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,334 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,335 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:43,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:43,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:43,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:43,337 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:43,337 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:43,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,338 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,338 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,340 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,341 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:43,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:43,341 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,342 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:43,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:43,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,342 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:43,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321583350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,442 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:43,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:43,445 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:43,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:43,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:43,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,448 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:43,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:43,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,452 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,452 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,452 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8fbc361[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:43,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:43,453 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,455 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:43,455 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,455 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:43,457 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:43,458 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:43,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:43,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:43,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:43,458 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:43,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:43,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,464 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:43,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:43,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,464 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:43,464 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:43,464 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:43,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/47483acef8694a7aa00f591953c34d45 2023-04-01 03:58:43,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/6d1ab37e696e4eafac6b82c0b4c66582 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582 2023-04-01 03:58:43,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582, entries=200, sequenceid=901, filesize=14.3 K 2023-04-01 03:58:43,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/8f42be692e304bf59e8bec128d2eb010 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010 2023-04-01 03:58:43,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010, entries=150, sequenceid=901, filesize=11.9 K 2023-04-01 03:58:43,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/47483acef8694a7aa00f591953c34d45 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45 2023-04-01 03:58:43,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45, entries=150, sequenceid=901, filesize=11.9 K 2023-04-01 03:58:43,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 64d06c34ee69fd6096a8db67850840d6 in 853ms, sequenceid=901, compaction requested=true 2023-04-01 03:58:43,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:43,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:43,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 03:58:43,535 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:43,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:43,535 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:43,536 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,536 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,536 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f4bad75ecbd942e698a09e58b489d617, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=39.0 K 2023-04-01 03:58:43,536 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b1cbd4e6548b4fd78151e271f5b2fed5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.7 K 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting f4bad75ecbd942e698a09e58b489d617, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520932 2023-04-01 03:58:43,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b1cbd4e6548b4fd78151e271f5b2fed5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520932 2023-04-01 03:58:43,537 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 862991d2c0794d1c9ba161e9254d6567, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680321521311 2023-04-01 03:58:43,537 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 050b849b312e4729af702eaf188522c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680321521311 2023-04-01 03:58:43,537 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6d1ab37e696e4eafac6b82c0b4c66582, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522020 2023-04-01 03:58:43,537 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 8f42be692e304bf59e8bec128d2eb010, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522052 2023-04-01 03:58:43,541 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#471 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:43,542 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:43,554 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/4e092fc8c3d54a3f84b5f594949ad96f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/4e092fc8c3d54a3f84b5f594949ad96f 2023-04-01 03:58:43,558 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 4e092fc8c3d54a3f84b5f594949ad96f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:43,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:43,558 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321523535; duration=0sec 2023-04-01 03:58:43,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:43,558 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:43,559 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:43,559 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:43,559 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,559 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/46e98d70eb52482f96f486a2ad22a37d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.7 K 2023-04-01 03:58:43,559 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 46e98d70eb52482f96f486a2ad22a37d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=865, earliestPutTs=1680321520932 2023-04-01 03:58:43,559 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ce981187733642aba942618d70e9b927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1680321521311 2023-04-01 03:58:43,560 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 47483acef8694a7aa00f591953c34d45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522052 2023-04-01 03:58:43,564 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:43,565 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:43,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:43,567 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:43,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:43,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:43,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,567 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,569 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:43,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:43,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,569 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:43,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:43,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:43,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:43,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:43,571 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:43,574 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:43,574 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. started... 2023-04-01 03:58:43,574 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:58:43,574 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:43,574 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:43,578 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/7c5165429c7a4a92b40670ad3fff3dc1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7c5165429c7a4a92b40670ad3fff3dc1 2023-04-01 03:58:43,582 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 7c5165429c7a4a92b40670ad3fff3dc1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:43,582 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:43,583 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321523535; duration=0sec 2023-04-01 03:58:43,583 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:43,583 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=915 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/7cdcf00273ee4d15aa43642f5473f081 2023-04-01 03:58:43,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:43,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. as already flushing 2023-04-01 03:58:43,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321583756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321583758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50526 deadline: 1680321583857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321583862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321583863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:43,952 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/f92b4e04cf41410b964ad2e74c7967ea as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f92b4e04cf41410b964ad2e74c7967ea 2023-04-01 03:58:43,959 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into f92b4e04cf41410b964ad2e74c7967ea(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:43,959 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:43,959 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321523535; duration=0sec 2023-04-01 03:58:43,959 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:43,992 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=915 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/aae76766974049419fdfd5a4b57a58aa 2023-04-01 03:58:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321584069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321584069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,314 DEBUG [Thread-2922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73892d82 to 127.0.0.1:50457 2023-04-01 03:58:44,314 DEBUG [Thread-2922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,316 DEBUG [Thread-2924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6cee7d6e to 127.0.0.1:50457 2023-04-01 03:58:44,316 DEBUG [Thread-2924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,317 DEBUG [Thread-2928] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22e3658a to 127.0.0.1:50457 2023-04-01 03:58:44,317 DEBUG [Thread-2928] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,318 DEBUG [Thread-2926] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5f8cb3 to 127.0.0.1:50457 2023-04-01 03:58:44,318 DEBUG [Thread-2926] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,320 DEBUG [Thread-2930] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b543524 to 127.0.0.1:50457 2023-04-01 03:58:44,320 DEBUG [Thread-2930] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50482 deadline: 1680321584376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50504 deadline: 1680321584378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,408 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=915 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/e306eda7d5294404aefc961828a5ba3e 2023-04-01 03:58:44,416 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/7cdcf00273ee4d15aa43642f5473f081 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081 2023-04-01 03:58:44,423 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081, entries=150, sequenceid=915, filesize=11.9 K 2023-04-01 03:58:44,424 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/aae76766974049419fdfd5a4b57a58aa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa 2023-04-01 03:58:44,431 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa, entries=150, sequenceid=915, filesize=11.9 K 2023-04-01 03:58:44,432 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/e306eda7d5294404aefc961828a5ba3e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e 2023-04-01 03:58:44,438 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e, entries=150, sequenceid=915, filesize=11.9 K 2023-04-01 03:58:44,438 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 64d06c34ee69fd6096a8db67850840d6 in 864ms, sequenceid=915, compaction requested=false 2023-04-01 03:58:44,438 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:44,439 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:44,439 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:44,439 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:44,439 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,439 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:44,439 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:44,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,441 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:44,441 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,441 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:44,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:44,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5927fd42[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:44,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:44,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:44,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:44,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:44,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,445 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:44,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:44,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:44,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,446 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:44,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:44,450 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:44,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:44,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:44,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:44,450 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:44,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:44,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:44,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:44,454 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,456 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:44,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:44,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:44,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:44,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:44,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:44,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:44,461 DEBUG [Thread-2921] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:44,461 DEBUG [Thread-2921] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:44,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:44,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:44,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,461 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:44,462 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:44,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:44,561 DEBUG [Thread-2921] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:44,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:44,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:44,871 DEBUG [Thread-2917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6cec010c to 127.0.0.1:50457 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:44,871 DEBUG [Thread-2917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:44,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:44,880 DEBUG [Thread-2919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e65fce to 127.0.0.1:50457 2023-04-01 03:58:44,880 DEBUG [Thread-2919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:44,880 DEBUG [Thread-2911] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2beab5e2 to 127.0.0.1:50457 2023-04-01 03:58:44,880 DEBUG [Thread-2911] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:45,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/306e082d4ac34028884065d5e7eb1926 2023-04-01 03:58:45,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/afee9a2862e141e18dcd38ab755a5e97 2023-04-01 03:58:46,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=937 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/08ae033407d44536a1735882cc7c0a4b 2023-04-01 03:58:46,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/306e082d4ac34028884065d5e7eb1926 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926 2023-04-01 03:58:46,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926, entries=150, sequenceid=937, filesize=11.9 K 2023-04-01 03:58:46,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/afee9a2862e141e18dcd38ab755a5e97 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97 2023-04-01 03:58:46,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97, entries=150, sequenceid=937, filesize=11.9 K 2023-04-01 03:58:46,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/08ae033407d44536a1735882cc7c0a4b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b 2023-04-01 03:58:46,112 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b, entries=150, sequenceid=937, filesize=11.9 K 2023-04-01 03:58:46,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=13.42 KB/13740 for 64d06c34ee69fd6096a8db67850840d6 in 1241ms, sequenceid=937, compaction requested=true 2023-04-01 03:58:46,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:46,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:46,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:46,113 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:46,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:46,113 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:46,113 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:46,113 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/A is initiating minor compaction (all files) 2023-04-01 03:58:46,113 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:46,114 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/A in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/B is initiating minor compaction (all files) 2023-04-01 03:58:46,114 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f92b4e04cf41410b964ad2e74c7967ea, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.8 K 2023-04-01 03:58:46,114 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/B in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:46,114 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/4e092fc8c3d54a3f84b5f594949ad96f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.8 K 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f92b4e04cf41410b964ad2e74c7967ea, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522052 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 4e092fc8c3d54a3f84b5f594949ad96f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522052 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7cdcf00273ee4d15aa43642f5473f081, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1680321522708 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting aae76766974049419fdfd5a4b57a58aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1680321522708 2023-04-01 03:58:46,114 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 306e082d4ac34028884065d5e7eb1926, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1680321523744 2023-04-01 03:58:46,115 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting afee9a2862e141e18dcd38ab755a5e97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1680321523744 2023-04-01 03:58:46,120 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#A#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:46,123 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:46,129 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/3568667d41ce45b7a003b0421bf59c0b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/3568667d41ce45b7a003b0421bf59c0b 2023-04-01 03:58:46,132 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/B of 64d06c34ee69fd6096a8db67850840d6 into 3568667d41ce45b7a003b0421bf59c0b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:46,132 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:46,132 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/B, priority=13, startTime=1680321526112; duration=0sec 2023-04-01 03:58:46,132 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:46,132 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:46,133 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:46,133 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 64d06c34ee69fd6096a8db67850840d6/C is initiating minor compaction (all files) 2023-04-01 03:58:46,133 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 64d06c34ee69fd6096a8db67850840d6/C in TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:46,133 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7c5165429c7a4a92b40670ad3fff3dc1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp, totalSize=36.8 K 2023-04-01 03:58:46,133 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7c5165429c7a4a92b40670ad3fff3dc1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1680321522052 2023-04-01 03:58:46,134 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e306eda7d5294404aefc961828a5ba3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1680321522708 2023-04-01 03:58:46,134 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 08ae033407d44536a1735882cc7c0a4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=937, earliestPutTs=1680321523744 2023-04-01 03:58:46,139 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 64d06c34ee69fd6096a8db67850840d6#C#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:46,533 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/f69953f696c54071bb4114cc2d232654 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f69953f696c54071bb4114cc2d232654 2023-04-01 03:58:46,540 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/A of 64d06c34ee69fd6096a8db67850840d6 into f69953f696c54071bb4114cc2d232654(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:46,541 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:46,541 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/A, priority=13, startTime=1680321526112; duration=0sec 2023-04-01 03:58:46,541 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:46,549 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/45b555239d2244c09e9334856e75cc48 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/45b555239d2244c09e9334856e75cc48 2023-04-01 03:58:46,554 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 64d06c34ee69fd6096a8db67850840d6/C of 64d06c34ee69fd6096a8db67850840d6 into 45b555239d2244c09e9334856e75cc48(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:46,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:46,554 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6., storeName=64d06c34ee69fd6096a8db67850840d6/C, priority=13, startTime=1680321526113; duration=0sec 2023-04-01 03:58:46,554 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:48,940 DEBUG [Thread-2913] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x495719a9 to 127.0.0.1:50457 2023-04-01 03:58:48,940 DEBUG [Thread-2913] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:49,003 DEBUG [Thread-2915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6eae89d2 to 127.0.0.1:50457 2023-04-01 03:58:49,003 DEBUG [Thread-2915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 287 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 183 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1845 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5535 rows 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1869 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5607 rows 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1867 2023-04-01 03:58:49,004 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5593 rows 2023-04-01 03:58:49,005 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-01 03:58:49,005 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5569 rows 2023-04-01 03:58:49,005 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-04-01 03:58:49,005 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(403): verified 5520 rows 2023-04-01 03:58:49,005 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:58:49,005 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1262233b to 127.0.0.1:50457 2023-04-01 03:58:49,005 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:58:49,009 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:58:49,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:58:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:49,013 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321529013"}]},"ts":"1680321529013"} 2023-04-01 03:58:49,014 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:58:49,109 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:58:49,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, UNASSIGN}] 2023-04-01 03:58:49,110 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, UNASSIGN 2023-04-01 03:58:49,111 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=64d06c34ee69fd6096a8db67850840d6, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:49,111 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321529111"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321529111"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321529111"}]},"ts":"1680321529111"} 2023-04-01 03:58:49,112 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:49,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 64d06c34ee69fd6096a8db67850840d6, disabling compactions & flushes 2023-04-01 03:58:49,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. after waiting 0 ms 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:49,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 64d06c34ee69fd6096a8db67850840d6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=A 2023-04-01 03:58:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:49,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=B 2023-04-01 03:58:49,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:49,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 64d06c34ee69fd6096a8db67850840d6, store=C 2023-04-01 03:58:49,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:49,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:49,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/64ce62423b98480ab444e06c746dbafa 2023-04-01 03:58:50,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/fe6caf34e96149048dd8fc6ae60ec850 2023-04-01 03:58:50,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:50,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=947 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/8285ce1c74dc4b3ab7b2020803cbc761 2023-04-01 03:58:50,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/A/64ce62423b98480ab444e06c746dbafa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/64ce62423b98480ab444e06c746dbafa 2023-04-01 03:58:50,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/64ce62423b98480ab444e06c746dbafa, entries=50, sequenceid=947, filesize=7.1 K 2023-04-01 03:58:50,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/B/fe6caf34e96149048dd8fc6ae60ec850 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fe6caf34e96149048dd8fc6ae60ec850 2023-04-01 03:58:50,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fe6caf34e96149048dd8fc6ae60ec850, entries=50, sequenceid=947, filesize=7.1 K 2023-04-01 03:58:50,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/.tmp/C/8285ce1c74dc4b3ab7b2020803cbc761 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8285ce1c74dc4b3ab7b2020803cbc761 2023-04-01 03:58:50,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8285ce1c74dc4b3ab7b2020803cbc761, entries=50, sequenceid=947, filesize=7.1 K 2023-04-01 03:58:50,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 64d06c34ee69fd6096a8db67850840d6 in 1238ms, sequenceid=947, compaction requested=false 2023-04-01 03:58:50,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f911af8a28604018afb17984d6216511, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/614425eb901b433a9ff8104993841c03, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/253d6d20838c4246a3cba26936087ea2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/1c2d107021164c2ebda8a760d4863e82, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/35f2b2e6e8af4ea0917a6f3157a55439, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d15df2f3afbc45ef80b620cc7ba8a7cd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/29e5689510d44cd2bbc926707473191b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f4bad75ecbd942e698a09e58b489d617, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f92b4e04cf41410b964ad2e74c7967ea, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926] to archive 2023-04-01 03:58:50,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:50,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/a51984d27f7c4d348eae9c8ee006fb29 2023-04-01 03:58:50,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/93ffe706292a403b9044ff6aacd278cd 2023-04-01 03:58:50,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c6013b9a54e642b0969aaa9f7a2d2e11 2023-04-01 03:58:50,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/18455d2efbcc4b41b0598046f363a885 2023-04-01 03:58:50,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f911af8a28604018afb17984d6216511 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f911af8a28604018afb17984d6216511 2023-04-01 03:58:50,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf3dac1592334c22a130eb5e5869c378 2023-04-01 03:58:50,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ca4a69a702974413a06f4754b3443991 2023-04-01 03:58:50,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/614425eb901b433a9ff8104993841c03 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/614425eb901b433a9ff8104993841c03 2023-04-01 03:58:50,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d38a2fdbcb874fdb8b9c8361b506d465 2023-04-01 03:58:50,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/bf1d0c65d54c4dc99aedf754d3f1ddf1 2023-04-01 03:58:50,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/253d6d20838c4246a3cba26936087ea2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/253d6d20838c4246a3cba26936087ea2 2023-04-01 03:58:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/70b007854d974db3bb9d74526683f619 2023-04-01 03:58:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4feaf550a2a648e3bda7a81d243aa838 2023-04-01 03:58:50,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/5edc4697f58f464f84b37fb7c52a5859 2023-04-01 03:58:50,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/391dd0d5438e41c1a297554a2e347aa6 2023-04-01 03:58:50,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/1c2d107021164c2ebda8a760d4863e82 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/1c2d107021164c2ebda8a760d4863e82 2023-04-01 03:58:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/323bd02b06ac4ca5a60b77246cbfe989 2023-04-01 03:58:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/34087fa98b9d4df5b1eadd012e1d8481 2023-04-01 03:58:50,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/35f2b2e6e8af4ea0917a6f3157a55439 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/35f2b2e6e8af4ea0917a6f3157a55439 2023-04-01 03:58:50,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/9e04c93cd7844179a9671dbe69f473a6 2023-04-01 03:58:50,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/fc88d049caba46e0bdcbdeadb0e54d44 2023-04-01 03:58:50,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d15df2f3afbc45ef80b620cc7ba8a7cd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/d15df2f3afbc45ef80b620cc7ba8a7cd 2023-04-01 03:58:50,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b075494102f94e56a53f8a89ddd9982a 2023-04-01 03:58:50,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/b1933ab2f16c4a4586c9bafab667c1eb 2023-04-01 03:58:50,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/c33e167c3db04a3b8c14c0290275d229 2023-04-01 03:58:50,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/29e5689510d44cd2bbc926707473191b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/29e5689510d44cd2bbc926707473191b 2023-04-01 03:58:50,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6c0dadaf2e024bd1affb04518172a454 2023-04-01 03:58:50,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/ebe1769e8693456a92e8e0ca36024b2a 2023-04-01 03:58:50,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/4b27f6c0dbe346a7a3e1030ee017a3d6 2023-04-01 03:58:50,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f4bad75ecbd942e698a09e58b489d617 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f4bad75ecbd942e698a09e58b489d617 2023-04-01 03:58:50,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/862991d2c0794d1c9ba161e9254d6567 2023-04-01 03:58:50,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/6d1ab37e696e4eafac6b82c0b4c66582 2023-04-01 03:58:50,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f92b4e04cf41410b964ad2e74c7967ea to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f92b4e04cf41410b964ad2e74c7967ea 2023-04-01 03:58:50,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/7cdcf00273ee4d15aa43642f5473f081 2023-04-01 03:58:50,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/306e082d4ac34028884065d5e7eb1926 2023-04-01 03:58:50,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/db1b7f0f1a644f74a51ea3373a26e049, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/99e1dacbc0364a4abba45690fdd165c0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/e17f0baef29e4568b44b8251a778d956, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/703d0735245746c0a740984f1936d256, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6bec20ff12ac4f6c9ed46e7a700855c9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/df0e1a06a6f542099b7cbaa9d4e17d91, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2f26bb2039604c2cb18890ce4e5a45d2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b1cbd4e6548b4fd78151e271f5b2fed5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/4e092fc8c3d54a3f84b5f594949ad96f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97] to archive 2023-04-01 03:58:50,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:50,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b0355dae33eb4c4b9898ccb860e07135 2023-04-01 03:58:50,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/93ed01385f3c431989258c593efe85c4 2023-04-01 03:58:50,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b8a3f3d8ac28491288b39b79297a134a 2023-04-01 03:58:50,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/db1b7f0f1a644f74a51ea3373a26e049 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/db1b7f0f1a644f74a51ea3373a26e049 2023-04-01 03:58:50,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/84706160fb1c4d29a12c79a94083728c 2023-04-01 03:58:50,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/1f709ccc15d7475e89b8e014476d896a 2023-04-01 03:58:50,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/99e1dacbc0364a4abba45690fdd165c0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/99e1dacbc0364a4abba45690fdd165c0 2023-04-01 03:58:50,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b110582d6a5640d58a4909a7e0c64541 2023-04-01 03:58:50,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/adeba511fd5c4f30829a79f33d77951f 2023-04-01 03:58:50,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/e17f0baef29e4568b44b8251a778d956 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/e17f0baef29e4568b44b8251a778d956 2023-04-01 03:58:50,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/68aed5c09da5484781afa34f433a073e 2023-04-01 03:58:50,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2d688dce30634d4eb0ed83d4bcfe2a3a 2023-04-01 03:58:50,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/449bcbb36fd4450c938a695949ea7033 2023-04-01 03:58:50,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/cc7496cbd3314a70818ad881c1b1b30f 2023-04-01 03:58:50,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/703d0735245746c0a740984f1936d256 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/703d0735245746c0a740984f1936d256 2023-04-01 03:58:50,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/46f7a7dff3694358aacafdf96b457361 2023-04-01 03:58:50,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/5f4fc4628158457bb8bf916c519855ce 2023-04-01 03:58:50,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6bec20ff12ac4f6c9ed46e7a700855c9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6bec20ff12ac4f6c9ed46e7a700855c9 2023-04-01 03:58:50,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/617fb97f31e5400d9677af0a0370d848 2023-04-01 03:58:50,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/de16ae39d46c49da8d57f589edd81b8a 2023-04-01 03:58:50,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/df0e1a06a6f542099b7cbaa9d4e17d91 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/df0e1a06a6f542099b7cbaa9d4e17d91 2023-04-01 03:58:50,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fca9db45b6054bfaba7818fc9ebd9092 2023-04-01 03:58:50,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2af4df768acd47a0860ee49d10ea26e5 2023-04-01 03:58:50,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/0ee5061e8adc4c28ab5f797f9bab73b6 2023-04-01 03:58:50,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2f26bb2039604c2cb18890ce4e5a45d2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/2f26bb2039604c2cb18890ce4e5a45d2 2023-04-01 03:58:50,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/6f34289ed13f434db362202bbd109bdf 2023-04-01 03:58:50,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/bd7de73e7bc24546b7f37871f1702db4 2023-04-01 03:58:50,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/59bd385adc0d4edf9533d93c2b65c80c 2023-04-01 03:58:50,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b1cbd4e6548b4fd78151e271f5b2fed5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/b1cbd4e6548b4fd78151e271f5b2fed5 2023-04-01 03:58:50,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/53a0671a01c340cf8cab274dfc1b09f3 2023-04-01 03:58:50,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/050b849b312e4729af702eaf188522c1 2023-04-01 03:58:50,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/4e092fc8c3d54a3f84b5f594949ad96f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/4e092fc8c3d54a3f84b5f594949ad96f 2023-04-01 03:58:50,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/8f42be692e304bf59e8bec128d2eb010 2023-04-01 03:58:50,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/aae76766974049419fdfd5a4b57a58aa 2023-04-01 03:58:50,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/afee9a2862e141e18dcd38ab755a5e97 2023-04-01 03:58:50,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/224cb3cc0f5946d195c21d0ec5002e85, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/6fdebfdd25394b538412e25533493a1e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c236152eed164695ac7914cda396dea7, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8cb064192139497f997a807721388d86, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e1f1caa773c743488e022923456b49ff, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2e0ee72066564777947a54c1dd951d44, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/278794eaecd245d9805b8a725eb6c614, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/46e98d70eb52482f96f486a2ad22a37d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7c5165429c7a4a92b40670ad3fff3dc1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b] to archive 2023-04-01 03:58:50,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:58:50,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/012a27f84f084531903b07a54dc48feb 2023-04-01 03:58:50,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d5cb16181f004901a33588cf12cef4e9 2023-04-01 03:58:50,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/44c8e9fb0b9e496bbfa6f998d98a40fe 2023-04-01 03:58:50,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/224cb3cc0f5946d195c21d0ec5002e85 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/224cb3cc0f5946d195c21d0ec5002e85 2023-04-01 03:58:50,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2fe8fd729e8b407ba2a463d169ab6ee2 2023-04-01 03:58:50,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cb8d69d64d8a4245be8db4a3c3a17933 2023-04-01 03:58:50,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/6fdebfdd25394b538412e25533493a1e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/6fdebfdd25394b538412e25533493a1e 2023-04-01 03:58:50,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e962b24b0a214dc1b54e8ca4755b16f9 2023-04-01 03:58:50,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/40caf61bf5404fd7ba043acf5d197449 2023-04-01 03:58:50,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c236152eed164695ac7914cda396dea7 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c236152eed164695ac7914cda396dea7 2023-04-01 03:58:50,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/037d35e9f6f2484bae5337e0823d2503 2023-04-01 03:58:50,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7f1f507b4d094db8b7f8b698b32a054f 2023-04-01 03:58:50,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/90fef2e9368c4b83a8ace126815b3037 2023-04-01 03:58:50,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/c99a540de9a244588cf26636713adc97 2023-04-01 03:58:50,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8cb064192139497f997a807721388d86 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8cb064192139497f997a807721388d86 2023-04-01 03:58:50,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/d93ae3913b0948899e6808c303293183 2023-04-01 03:58:50,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8fde6e8bd3b74a46ad6e94aa5865644e 2023-04-01 03:58:50,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e1f1caa773c743488e022923456b49ff to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e1f1caa773c743488e022923456b49ff 2023-04-01 03:58:50,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/74d71607aa9e4b7b85135c8aaac07caf 2023-04-01 03:58:50,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/9f5a9a7e276e47ff8b4183c3d0d679f6 2023-04-01 03:58:50,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2e0ee72066564777947a54c1dd951d44 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/2e0ee72066564777947a54c1dd951d44 2023-04-01 03:58:50,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/f4584d8180954ef3b6c93e802240ef4a 2023-04-01 03:58:50,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/3b7c88d98426409ea4a846285d743779 2023-04-01 03:58:50,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/54607a65e62e4201a9af0760aae67b50 2023-04-01 03:58:50,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/278794eaecd245d9805b8a725eb6c614 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/278794eaecd245d9805b8a725eb6c614 2023-04-01 03:58:50,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/cd1cd282790945d1b80d345ac57d7bd0 2023-04-01 03:58:50,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/fad40ee75b404aa0bb2cbfb21f020460 2023-04-01 03:58:50,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/0af13bde649f439bbb182d9003bedad2 2023-04-01 03:58:50,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/46e98d70eb52482f96f486a2ad22a37d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/46e98d70eb52482f96f486a2ad22a37d 2023-04-01 03:58:50,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/117ef7a81c5c4d2b9f4d7ecd1011c972 2023-04-01 03:58:50,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/ce981187733642aba942618d70e9b927 2023-04-01 03:58:50,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7c5165429c7a4a92b40670ad3fff3dc1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/7c5165429c7a4a92b40670ad3fff3dc1 2023-04-01 03:58:50,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/47483acef8694a7aa00f591953c34d45 2023-04-01 03:58:50,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/e306eda7d5294404aefc961828a5ba3e 2023-04-01 03:58:50,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/08ae033407d44536a1735882cc7c0a4b 2023-04-01 03:58:50,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/recovered.edits/950.seqid, newMaxSeqId=950, maxSeqId=1 2023-04-01 03:58:50,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6. 2023-04-01 03:58:50,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 64d06c34ee69fd6096a8db67850840d6: 2023-04-01 03:58:50,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:50,648 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=64d06c34ee69fd6096a8db67850840d6, regionState=CLOSED 2023-04-01 03:58:50,648 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321530648"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321530648"}]},"ts":"1680321530648"} 2023-04-01 03:58:50,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-01 03:58:50,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 64d06c34ee69fd6096a8db67850840d6, server=jenkins-hbase20.apache.org,36251,1680321375816 in 1.5370 sec 2023-04-01 03:58:50,651 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-01 03:58:50,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=64d06c34ee69fd6096a8db67850840d6, UNASSIGN in 1.5400 sec 2023-04-01 03:58:50,652 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321530652"}]},"ts":"1680321530652"} 2023-04-01 03:58:50,653 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:58:50,654 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:58:50,655 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6450 sec 2023-04-01 03:58:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 03:58:51,117 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-01 03:58:51,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:58:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,120 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-01 03:58:51,120 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,124 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:51,126 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/recovered.edits] 2023-04-01 03:58:51,128 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/64ce62423b98480ab444e06c746dbafa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/64ce62423b98480ab444e06c746dbafa 2023-04-01 03:58:51,130 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f69953f696c54071bb4114cc2d232654 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/A/f69953f696c54071bb4114cc2d232654 2023-04-01 03:58:51,132 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/3568667d41ce45b7a003b0421bf59c0b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/3568667d41ce45b7a003b0421bf59c0b 2023-04-01 03:58:51,133 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fe6caf34e96149048dd8fc6ae60ec850 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/B/fe6caf34e96149048dd8fc6ae60ec850 2023-04-01 03:58:51,134 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/45b555239d2244c09e9334856e75cc48 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/45b555239d2244c09e9334856e75cc48 2023-04-01 03:58:51,135 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8285ce1c74dc4b3ab7b2020803cbc761 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/C/8285ce1c74dc4b3ab7b2020803cbc761 2023-04-01 03:58:51,137 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/recovered.edits/950.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6/recovered.edits/950.seqid 2023-04-01 03:58:51,138 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/64d06c34ee69fd6096a8db67850840d6 2023-04-01 03:58:51,138 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:58:51,139 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,140 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:58:51,142 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:58:51,142 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,143 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:58:51,143 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321531143"}]},"ts":"9223372036854775807"} 2023-04-01 03:58:51,144 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:58:51,144 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 64d06c34ee69fd6096a8db67850840d6, NAME => 'TestAcidGuarantees,,1680321502992.64d06c34ee69fd6096a8db67850840d6.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:58:51,144 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:58:51,144 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321531144"}]},"ts":"9223372036854775807"} 2023-04-01 03:58:51,145 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:58:51,147 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-04-01 03:58:51,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-01 03:58:51,222 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-01 03:58:51,238 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 230), OpenFileDescriptor=547 (was 532) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=625 (was 606) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=9044 (was 8128) - AvailableMemoryMB LEAK? - 2023-04-01 03:58:51,250 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=625, ProcessCount=169, AvailableMemoryMB=9044 2023-04-01 03:58:51,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:58:51,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:58:51,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:51,255 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 03:58:51,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-01 03:58:51,255 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 03:58:51,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 03:58:51,256 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,257 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a empty. 2023-04-01 03:58:51,257 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,257 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:58:51,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 03:58:51,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 03:58:51,665 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:58:51,666 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 08f6a4fd8a5792a8aca3a5d3a16a6b7a, NAME => 'TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp 2023-04-01 03:58:51,675 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:58:51,676 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 08f6a4fd8a5792a8aca3a5d3a16a6b7a, disabling compactions & flushes 2023-04-01 03:58:51,676 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:51,676 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:51,676 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. after waiting 0 ms 2023-04-01 03:58:51,676 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:51,676 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:51,676 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:51,679 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 03:58:51,680 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321531680"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321531680"}]},"ts":"1680321531680"} 2023-04-01 03:58:51,681 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 03:58:51,681 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 03:58:51,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321531681"}]},"ts":"1680321531681"} 2023-04-01 03:58:51,682 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 03:58:51,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, ASSIGN}] 2023-04-01 03:58:51,686 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, ASSIGN 2023-04-01 03:58:51,687 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=false 2023-04-01 03:58:51,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:51,838 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321531838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321531838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321531838"}]},"ts":"1680321531838"} 2023-04-01 03:58:51,840 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:51,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 03:58:51,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:51,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08f6a4fd8a5792a8aca3a5d3a16a6b7a, NAME => 'TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:58:51,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:58:51,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,995 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,996 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A 2023-04-01 03:58:51,996 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A 2023-04-01 03:58:51,996 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:51,996 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName A 2023-04-01 03:58:51,997 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:51,997 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,997 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B 2023-04-01 03:58:51,998 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B 2023-04-01 03:58:51,998 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:51,998 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName B 2023-04-01 03:58:51,998 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:51,998 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:51,999 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C 2023-04-01 03:58:51,999 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C 2023-04-01 03:58:51,999 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:52,000 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName C 2023-04-01 03:58:52,000 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:52,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:52,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:52,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:58:52,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:52,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 03:58:52,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08f6a4fd8a5792a8aca3a5d3a16a6b7a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61052714, jitterRate=-0.09024366736412048}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:58:52,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:52,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., pid=56, masterSystemTime=1680321531991 2023-04-01 03:58:52,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,007 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:52,007 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321532007"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321532007"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321532007"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321532007"}]},"ts":"1680321532007"} 2023-04-01 03:58:52,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-01 03:58:52,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 in 168 msec 2023-04-01 03:58:52,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-01 03:58:52,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, ASSIGN in 324 msec 2023-04-01 03:58:52,010 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 03:58:52,010 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321532010"}]},"ts":"1680321532010"} 2023-04-01 03:58:52,011 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 03:58:52,014 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 03:58:52,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 762 msec 2023-04-01 03:58:52,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 03:58:52,363 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-01 03:58:52,364 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x2da434f4 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:52,368 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ca54bf4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:52,369 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:52,370 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:52,371 DEBUG [Listener at localhost.localdomain/39075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 03:58:52,373 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 03:58:52,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 03:58:52,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 03:58:52,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 03:58:52,785 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:58:52,786 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 03:58:52,786 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 03:58:52,788 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 03:58:52,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, REOPEN/MOVE}] 2023-04-01 03:58:52,791 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, REOPEN/MOVE 2023-04-01 03:58:52,791 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:52,792 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321532791"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321532791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321532791"}]},"ts":"1680321532791"} 2023-04-01 03:58:52,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:52,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:52,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08f6a4fd8a5792a8aca3a5d3a16a6b7a, disabling compactions & flushes 2023-04-01 03:58:52,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. after waiting 0 ms 2023-04-01 03:58:52,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 03:58:52,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:52,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:52,949 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 08f6a4fd8a5792a8aca3a5d3a16a6b7a to self. 2023-04-01 03:58:52,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:52,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=CLOSED 2023-04-01 03:58:52,950 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321532950"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321532950"}]},"ts":"1680321532950"} 2023-04-01 03:58:52,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-01 03:58:52,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 in 159 msec 2023-04-01 03:58:52,953 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36251,1680321375816; forceNewPlan=false, retain=true 2023-04-01 03:58:53,103 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,104 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321533103"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321533103"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321533103"}]},"ts":"1680321533103"} 2023-04-01 03:58:53,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:58:53,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08f6a4fd8a5792a8aca3a5d3a16a6b7a, NAME => 'TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.', STARTKEY => '', ENDKEY => ''} 2023-04-01 03:58:53,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 03:58:53,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,261 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,263 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A 2023-04-01 03:58:53,263 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A 2023-04-01 03:58:53,263 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:53,263 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName A 2023-04-01 03:58:53,265 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:53,265 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,266 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B 2023-04-01 03:58:53,266 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B 2023-04-01 03:58:53,266 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:53,266 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName B 2023-04-01 03:58:53,267 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:53,267 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,268 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C 2023-04-01 03:58:53,268 DEBUG [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C 2023-04-01 03:58:53,268 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 03:58:53,268 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f6a4fd8a5792a8aca3a5d3a16a6b7a columnFamilyName C 2023-04-01 03:58:53,269 INFO [StoreOpener-08f6a4fd8a5792a8aca3a5d3a16a6b7a-1] regionserver.HStore(310): Store=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 03:58:53,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 03:58:53,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08f6a4fd8a5792a8aca3a5d3a16a6b7a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69439443, jitterRate=0.03472833335399628}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 03:58:53,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:53,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., pid=61, masterSystemTime=1680321533256 2023-04-01 03:58:53,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,276 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,276 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321533276"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680321533276"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680321533276"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680321533276"}]},"ts":"1680321533276"} 2023-04-01 03:58:53,278 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-01 03:58:53,278 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 in 172 msec 2023-04-01 03:58:53,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-01 03:58:53,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, REOPEN/MOVE in 488 msec 2023-04-01 03:58:53,280 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-01 03:58:53,281 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-01 03:58:53,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-01 03:58:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-01 03:58:53,284 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x2145bb3e to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,289 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d4a3876, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,289 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x23c268ca to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,298 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fd2bbc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,298 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x29eda94b to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,305 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19591084, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,305 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x4dd8f6f2 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,310 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@93ba9f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,310 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x07e01249 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,317 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d050b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,318 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3c2e323a to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,323 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19576a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,323 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x09a9d023 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,328 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445b9cce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,329 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x53b986df to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,335 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14566c43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,336 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x3a953bfa to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,341 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2585cd09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,341 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(139): Connect 0x170f7240 to 127.0.0.1:50457 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 03:58:53,346 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d6c1b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 03:58:53,357 DEBUG [hconnection-0x7e6afb84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,357 DEBUG [hconnection-0x3e67a955-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,357 DEBUG [hconnection-0xd6681c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,358 DEBUG [hconnection-0xdcb2694-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,358 DEBUG [hconnection-0x33485837-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,363 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40562, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,363 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,363 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,363 DEBUG [hconnection-0x4b328644-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,363 DEBUG [hconnection-0x5ad0580a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,360 DEBUG [hconnection-0x63120b7f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,364 DEBUG [hconnection-0x744c214-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,366 DEBUG [hconnection-0x264b69c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 03:58:53,366 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,371 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,371 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,373 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40576, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,373 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,374 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,374 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 03:58:53,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:53,381 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:53,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:53,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:53,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:53,384 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 03:58:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:53,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:53,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:53,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:53,385 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:53,385 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,385 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:53,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:53,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:53,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:53,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:53,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:53,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:53,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:53,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:53,394 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:53,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:53,396 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,396 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,396 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,396 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:53,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63afc31f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:53,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:53,398 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321593395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321593395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321593397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,400 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:53,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:53,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,400 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:53,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321593399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321593399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,400 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:53,403 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:53,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,403 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:53,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:53,405 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:53,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:53,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:53,406 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:53,406 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:53,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,407 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,409 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:53,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:53,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:53,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,415 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:53,415 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:53,446 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040140a57b7123524826885b9fa0c143ef77 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040140a57b7123524826885b9fa0c143ef77 2023-04-01 03:58:53,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321593499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321593499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321593500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321593502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321593502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,515 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:53,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:53,518 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:53,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:53,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:53,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,520 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,521 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,521 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:53,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:53,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:53,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:53,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,523 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:53,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:53,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:53,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:53,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:53,527 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,527 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e5e4be0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:53,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:53,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,530 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:53,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:53,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:53,530 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:53,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:53,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:53,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:53,534 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:53,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:53,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:53,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:53,534 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:53,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,537 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,538 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:53,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:53,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,541 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:53,541 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:53,541 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,642 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:53,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:53,645 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:53,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:53,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:53,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,646 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:53,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:53,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:53,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:53,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:53,654 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,654 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:53,654 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:53,654 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:53,654 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:53,658 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,658 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,659 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:53,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e87b81[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:53,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:53,661 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,661 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:53,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:53,662 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,662 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:53,664 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:53,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:53,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:53,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:53,664 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:53,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,665 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,666 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:53,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,668 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:53,668 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:53,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321593702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321593703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321593704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:53,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321593705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321593706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,768 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:53,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:53,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:53,771 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:53,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:53,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:53,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,796 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:53,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:53,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:53,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:53,797 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,797 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:53,797 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:53,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:53,798 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,799 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:53,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec34a33[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:53,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:53,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:53,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:53,801 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:53,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:53,802 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:53,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:53,804 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:53,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:53,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:53,804 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:53,804 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:53,805 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,805 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,805 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,816 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:53,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,817 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:53,817 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:53,816 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:53,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,817 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,870 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7e0e1d359bb94c879a8157d89db8a311 2023-04-01 03:58:53,917 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:53,920 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:53,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:53,920 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:53,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,921 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,921 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,922 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,922 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:53,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:53,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:53,923 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,923 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:53,923 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:53,923 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:53,923 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:53,925 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,925 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:53,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e9cbcf[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:53,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:53,926 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,928 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:53,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:53,929 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:53,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,929 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:53,930 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:53,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:53,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:53,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:53,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:53,932 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:53,932 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:53,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:53,934 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:53,934 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,935 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,937 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:53,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:53,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:53,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:53,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:53,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:53,937 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:53,938 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:53,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:53,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:53,938 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321594006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321594006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321594007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321594008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321594010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,038 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,041 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,041 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,042 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,042 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,043 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,043 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,043 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,044 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,044 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,046 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,046 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc49f19[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,047 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,048 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,048 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,051 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,051 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,051 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,052 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,053 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,055 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,055 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,055 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,055 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,155 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,158 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,158 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,159 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,159 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,160 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,160 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,161 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,161 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,161 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,161 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,161 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,163 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,163 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,163 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,163 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143bc313[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,164 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,166 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,166 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,166 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,167 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,171 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,171 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,171 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,171 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,172 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,172 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,172 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,172 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,172 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,174 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,174 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,174 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,174 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,178 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,179 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,179 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,179 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,279 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/0f69f4e2ac6548c6b06570732135de86 2023-04-01 03:58:54,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,282 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,282 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,285 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,285 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,286 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,294 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,294 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,298 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,299 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60dd8175[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,300 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,300 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,300 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,300 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,301 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,301 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,301 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,303 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,303 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,303 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,304 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,304 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,305 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,306 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,307 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,307 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,307 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,307 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,407 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,412 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,412 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,421 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,422 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,422 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,423 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,423 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,423 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,423 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,423 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,433 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dd78912[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,434 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,435 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,435 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,436 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,436 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,437 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,439 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,440 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,442 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,444 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,446 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,446 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,446 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,446 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321594508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321594509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321594512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321594514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321594525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,546 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,549 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,551 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,552 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,552 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,552 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,552 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,552 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,554 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,554 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2610f9e6[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,555 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,555 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,557 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,557 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,557 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,559 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,560 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,560 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,563 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,564 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,565 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,565 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,566 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,566 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,566 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,666 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,670 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,670 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,672 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,673 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,674 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,675 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:54,675 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:54,676 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:54,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf9856c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:54,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:54,677 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:54,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:54,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:54,678 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:54,682 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:54,684 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:54,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:54,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:54,684 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:54,684 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:54,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,685 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,687 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,689 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:54,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:54,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:54,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:54,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:54,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:54,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,690 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:54,690 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:54,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/fdd3179c4a5a4b4d858b06753b55be36 2023-04-01 03:58:54,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7e0e1d359bb94c879a8157d89db8a311 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311 2023-04-01 03:58:54,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311, entries=150, sequenceid=16, filesize=25.1 K 2023-04-01 03:58:54,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/0f69f4e2ac6548c6b06570732135de86 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86 2023-04-01 03:58:54,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86, entries=150, sequenceid=16, filesize=11.6 K 2023-04-01 03:58:54,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/fdd3179c4a5a4b4d858b06753b55be36 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36 2023-04-01 03:58:54,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36, entries=150, sequenceid=16, filesize=11.6 K 2023-04-01 03:58:54,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1329ms, sequenceid=16, compaction requested=false 2023-04-01 03:58:54,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:54,790 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:54,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:54,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:54,793 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:54,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:54,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:54,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,796 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:54,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:54,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:54,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:54,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:54,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:54,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:54,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:54,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:54,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:54,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:54,797 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:54,797 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:54,797 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:54,798 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:54,798 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:55,212 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011332caedbd6548778d81ff4a345147b8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011332caedbd6548778d81ff4a345147b8 2023-04-01 03:58:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:55,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:55,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321595520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321595522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321595522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321595522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321595527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,615 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c6865f90269b4aa69dbf7771e0f8670d 2023-04-01 03:58:55,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321595623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321595625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321595625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321595824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321595827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:55,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321595828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,023 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/bc917ac4731e4b18a5d19a6cb0071916 2023-04-01 03:58:56,031 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/aa083c6cd9c6489ea0d443b2791d77ae 2023-04-01 03:58:56,036 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c6865f90269b4aa69dbf7771e0f8670d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d 2023-04-01 03:58:56,041 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d, entries=150, sequenceid=36, filesize=25.1 K 2023-04-01 03:58:56,041 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/bc917ac4731e4b18a5d19a6cb0071916 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916 2023-04-01 03:58:56,046 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916, entries=150, sequenceid=36, filesize=11.6 K 2023-04-01 03:58:56,047 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/aa083c6cd9c6489ea0d443b2791d77ae as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae 2023-04-01 03:58:56,049 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae, entries=150, sequenceid=36, filesize=11.6 K 2023-04-01 03:58:56,050 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1252ms, sequenceid=36, compaction requested=false 2023-04-01 03:58:56,050 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:56,050 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,050 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,052 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60a367af[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,054 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,056 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,056 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,056 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,056 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,073 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,077 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,077 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,077 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,101 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:56,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,103 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,103 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,103 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:56,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:56,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:56,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321596140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321596141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321596142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,157 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 03:58:56,203 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,206 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,208 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,209 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,209 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,212 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,214 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,214 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,233 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15dedb0[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,235 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,236 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,237 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,237 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,241 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,241 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321596244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321596244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321596244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,250 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,250 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,250 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,250 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,253 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,258 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,259 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,259 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,359 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,375 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,377 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,377 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,378 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,379 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,380 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,380 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,380 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,380 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,380 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,381 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,382 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2023fbb4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,386 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,386 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,387 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,387 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,388 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,388 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,389 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,392 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,392 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,392 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,395 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,397 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,401 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,401 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,402 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,402 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,402 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,402 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,402 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321596445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321596445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321596446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,502 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,505 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,505 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,506 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,506 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,507 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,508 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,508 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,508 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,509 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,509 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1d839[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,510 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,511 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,511 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,511 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,511 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,512 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,513 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,513 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,513 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,513 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,513 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,514 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,515 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,518 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,518 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,519 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,519 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,537 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040160a1fdb40b1a486896deefbcb652b889 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040160a1fdb40b1a486896deefbcb652b889 2023-04-01 03:58:56,619 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,634 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,634 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,634 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,634 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,635 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,635 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,636 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,638 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,643 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,643 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,644 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b344993[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,648 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,648 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,649 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,649 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,651 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,651 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,652 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,653 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,655 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,655 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,656 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321596749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321596750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:56,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321596750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,756 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,760 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,761 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,761 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,763 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ea0b28[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,765 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,765 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,766 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,766 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,767 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,767 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,769 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,769 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,771 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,771 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,772 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,774 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,774 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:56,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,775 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,775 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,775 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,875 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:56,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:56,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:56,882 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:56,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:56,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:56,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,884 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,885 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,886 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:56,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:56,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:56,887 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,887 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:56,887 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:56,887 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:56,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:56,889 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,889 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:56,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aabac7a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:56,895 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:56,895 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,896 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,896 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:56,896 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:56,896 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,897 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:56,897 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:56,898 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:56,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:56,900 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:56,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:56,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:56,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:56,900 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:56,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,901 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:56,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:56,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:56,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:56,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:56,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:56,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:56,906 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:56,906 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:56,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:56,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:56,952 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/567bb6066f44404cb472e826c37f1f6f 2023-04-01 03:58:56,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/e04d83af8a70445ba2832eeb30a2353a 2023-04-01 03:58:56,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/66486068c07f4d03bfe67a9a143f115f 2023-04-01 03:58:56,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/567bb6066f44404cb472e826c37f1f6f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f 2023-04-01 03:58:56,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f, entries=150, sequenceid=49, filesize=25.1 K 2023-04-01 03:58:56,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/e04d83af8a70445ba2832eeb30a2353a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a 2023-04-01 03:58:56,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a, entries=150, sequenceid=49, filesize=11.6 K 2023-04-01 03:58:56,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/66486068c07f4d03bfe67a9a143f115f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f 2023-04-01 03:58:57,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-01 03:58:57,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 875ms, sequenceid=49, compaction requested=true 2023-04-01 03:58:57,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:57,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:57,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:57,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:58:57,003 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:58:57,003 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:57,003 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:57,004 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=75.4 K 2023-04-01 03:58:57,004 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=34.8 K 2023-04-01 03:58:57,004 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7e0e1d359bb94c879a8157d89db8a311, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321533380 2023-04-01 03:58:57,004 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0f69f4e2ac6548c6b06570732135de86, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321533380 2023-04-01 03:58:57,004 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c6865f90269b4aa69dbf7771e0f8670d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680321533392 2023-04-01 03:58:57,004 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bc917ac4731e4b18a5d19a6cb0071916, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680321533392 2023-04-01 03:58:57,004 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 567bb6066f44404cb472e826c37f1f6f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:58:57,005 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e04d83af8a70445ba2832eeb30a2353a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:58:57,006 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:57,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:57,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:57,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:57,011 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:57,011 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:57,011 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:57,011 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:57,011 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,011 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:57,012 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:57,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,013 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#495 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:57,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:57,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:57,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:57,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:57,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:57,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:57,018 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:57,018 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:57,018 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:57,026 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:57,026 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:57,026 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:57,026 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:57,026 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 03:58:57,026 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:57,027 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:57,027 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:57,027 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:57,027 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:57,027 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:57,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:57,253 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:57,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321597264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321597265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321597265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321597368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321597368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321597369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,431 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/e8f1f56b9bf345aabdd778522c45ebf8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e8f1f56b9bf345aabdd778522c45ebf8 2023-04-01 03:58:57,434 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into e8f1f56b9bf345aabdd778522c45ebf8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:57,435 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:57,435 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=13, startTime=1680321537003; duration=0sec 2023-04-01 03:58:57,435 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:58:57,435 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:58:57,436 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:58:57,437 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:58:57,437 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:57,437 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=34.8 K 2023-04-01 03:58:57,438 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting fdd3179c4a5a4b4d858b06753b55be36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680321533380 2023-04-01 03:58:57,438 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting aa083c6cd9c6489ea0d443b2791d77ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680321533392 2023-04-01 03:58:57,439 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 66486068c07f4d03bfe67a9a143f115f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:58:57,443 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:58:57,446 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401dd420a2e771744db9bea5f7a276d287b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401dd420a2e771744db9bea5f7a276d287b 2023-04-01 03:58:57,450 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c1fdf73b8bf847c8909e55376d8c9778 2023-04-01 03:58:57,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321597539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,540 DEBUG [Thread-3647] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:57,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321597543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,546 DEBUG [Thread-3649] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:58:57,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321597571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321597571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321597572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,851 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/75b0ea63f1b547429e8efa9609eaf370 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/75b0ea63f1b547429e8efa9609eaf370 2023-04-01 03:58:57,856 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 75b0ea63f1b547429e8efa9609eaf370(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:58:57,856 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:57,856 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=13, startTime=1680321537003; duration=0sec 2023-04-01 03:58:57,857 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:57,858 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/16227592f8fc4adca4ca53a6cbd9f592 2023-04-01 03:58:57,867 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/e951a4a97657488eb8aafb6c6108d169 2023-04-01 03:58:57,871 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c1fdf73b8bf847c8909e55376d8c9778 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778 2023-04-01 03:58:57,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321597872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,875 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778, entries=150, sequenceid=68, filesize=25.1 K 2023-04-01 03:58:57,876 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/16227592f8fc4adca4ca53a6cbd9f592 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592 2023-04-01 03:58:57,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321597874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321597875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,880 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592, entries=150, sequenceid=68, filesize=11.6 K 2023-04-01 03:58:57,881 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/e951a4a97657488eb8aafb6c6108d169 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169 2023-04-01 03:58:57,884 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169, entries=150, sequenceid=68, filesize=11.6 K 2023-04-01 03:58:57,885 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 859ms, sequenceid=68, compaction requested=false 2023-04-01 03:58:57,885 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:57,885 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:57,885 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:57,885 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:57,885 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,885 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:57,885 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:57,887 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,887 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,888 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:57,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:57,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b5b906[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:57,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:57,889 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:57,891 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,891 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:57,893 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:57,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:57,897 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:57,897 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:57,897 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:57,897 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:57,897 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:57,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:57,900 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,900 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:57,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:57,905 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:57,905 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,905 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:57,905 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:57,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:57,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:57,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:57,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:57,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:57,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,005 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:58,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:58,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:58,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:58,008 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:58,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:58,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:58,012 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:58,012 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:58,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:58,013 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:58,013 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:58,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:58,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:58,014 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:58:58,262 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/b18e633b9de24abe95c264517b0531bd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/b18e633b9de24abe95c264517b0531bd 2023-04-01 03:58:58,264 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into b18e633b9de24abe95c264517b0531bd(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:58:58,264 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:58,265 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=13, startTime=1680321537002; duration=1sec 2023-04-01 03:58:58,265 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:58:58,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:58,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:58,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321598392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321598393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321598394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,431 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401756b71dd6dc24857bab7b13d9ae30ee9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401756b71dd6dc24857bab7b13d9ae30ee9 2023-04-01 03:58:58,434 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9ae9021e9a074042b5cdf0ae65d4b0a8 2023-04-01 03:58:58,442 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/6124690a6f064e628e8889c6b0e21c5a 2023-04-01 03:58:58,456 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4e93d7a06c144b57a3acc97584803a1c 2023-04-01 03:58:58,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9ae9021e9a074042b5cdf0ae65d4b0a8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8 2023-04-01 03:58:58,475 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8, entries=150, sequenceid=83, filesize=25.1 K 2023-04-01 03:58:58,475 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/6124690a6f064e628e8889c6b0e21c5a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a 2023-04-01 03:58:58,484 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a, entries=150, sequenceid=83, filesize=11.6 K 2023-04-01 03:58:58,485 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4e93d7a06c144b57a3acc97584803a1c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c 2023-04-01 03:58:58,488 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c, entries=150, sequenceid=83, filesize=11.6 K 2023-04-01 03:58:58,489 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 476ms, sequenceid=83, compaction requested=true 2023-04-01 03:58:58,489 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:58:58,489 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:58,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:58,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:58,489 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:58,492 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,492 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,492 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:58,493 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:58,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d9404d2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:58,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:58,495 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:58:58,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-01 03:58:58,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:58:58,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:58,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:58:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:58:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:58:58,508 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:58,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:58,508 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:58,516 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:58,516 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,516 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,516 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:58,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:58,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:58,517 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:58,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:58,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:58,517 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:58,517 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:58,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321598515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321598518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321598519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,525 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,527 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:58,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:58,534 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,534 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,535 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:58,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,535 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,535 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:58,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321598619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321598622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321598622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,635 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:58,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:58,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:58,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:58,638 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:58,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:58,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:58,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,638 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,640 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,640 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:58,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:58,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:58,641 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,641 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:58,641 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:58,641 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:58,641 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:58,643 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:58,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3862e303[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:58,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:58,645 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,645 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:58,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:58,647 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:58,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:58,649 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:58,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:58,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:58,649 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:58,649 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:58,650 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,651 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,652 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,654 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,654 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:58,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:58,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,655 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,655 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,655 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,755 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:58,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:58,758 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:58,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:58,758 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:58,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,760 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,760 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:58,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:58,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:58,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:58,762 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:58,762 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,763 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:58,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5d4009[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:58,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:58,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,764 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:58,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,764 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:58,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:58,764 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:58,765 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:58,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,765 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:58,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:58,767 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:58,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:58,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:58,767 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:58,767 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:58,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,768 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,769 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:58,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,770 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,770 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,770 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:58,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,770 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321598822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321598827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:58,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321598827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,870 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:58,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:58,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:58,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:58,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:58,873 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:58,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:58,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:58,874 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,874 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,875 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:58,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:58,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:58,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:58,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:58,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:58,877 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:58,877 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:58,879 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,879 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,880 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:58,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4629bd6d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:58,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:58,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:58,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:58,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,882 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:58,882 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:58,883 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,883 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,884 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:58,885 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:58,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:58,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:58,885 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:58,885 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:58,887 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,887 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:58,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,887 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,888 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,888 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:58,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:58,890 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:58,890 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:58,891 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,891 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:58,891 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:58,914 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015a5d7bf9726b4bffa686cd9a06684a43 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015a5d7bf9726b4bffa686cd9a06684a43 2023-04-01 03:58:58,991 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:58,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:58,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:58,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:58,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:58,994 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:58,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:58,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:58,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:58,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,996 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:58,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:58,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:58,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:58,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:58,996 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:58,997 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:58,997 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:58,997 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:58,997 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:58,997 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,001 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,002 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc693ee[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,002 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,003 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,003 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,003 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,004 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,004 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,005 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,005 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,005 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,006 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,006 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,006 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,007 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,008 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,009 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,009 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,009 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,109 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,112 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,112 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,112 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,112 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,113 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,113 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,115 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,115 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3413116e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,120 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,121 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,123 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,123 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,124 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321599124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,127 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,126 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,127 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,127 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,127 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321599131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321599131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,227 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,230 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,230 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,232 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,233 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,233 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,234 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,234 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,235 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11eb52ed[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,236 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,236 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,236 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,237 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,239 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,239 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,239 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,239 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,239 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,240 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,240 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,241 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,243 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,243 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,244 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,244 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,244 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,318 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9fc6722f0b72427c8d04a7690ea0e03f 2023-04-01 03:58:59,344 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,347 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,351 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,353 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,353 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a224768[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,357 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,357 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,358 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,358 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,358 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,360 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,360 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,360 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,361 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,361 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,362 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,363 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,363 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,364 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,364 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,364 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,464 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,467 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,469 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,469 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,470 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,471 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,471 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,472 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,472 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,473 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,473 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3554fe21[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,475 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,475 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,476 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,477 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,477 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,477 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,478 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,479 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,491 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,491 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,492 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,492 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,492 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,592 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,595 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,595 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,596 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,596 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,597 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,597 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,597 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,597 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,597 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,597 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,597 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,598 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,599 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,599 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e00184[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,601 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,601 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,601 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,602 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,602 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,604 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,604 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,604 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,604 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,604 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,604 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,604 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,604 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,605 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,605 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,607 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,607 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,607 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,607 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321599630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321599635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:58:59,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321599636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,707 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,710 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,710 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,710 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,710 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,710 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,710 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,711 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,712 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,712 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,712 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,712 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,713 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,713 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52d7446f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,715 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,715 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,716 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,718 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,718 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,719 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,720 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,721 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,723 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,723 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,723 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,723 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/e28412a411404725b294ee57a86d5641 2023-04-01 03:58:59,823 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,826 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,826 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,827 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,827 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,828 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,829 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,829 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,829 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,829 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,829 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:58:59,831 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,831 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:58:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:58:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac6505a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:58:59,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:58:59,834 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,836 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,836 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:58:59,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,836 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:58:59,836 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:58:59,836 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:58:59,838 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:58:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:58:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:58:59,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:58:59,841 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:58:59,841 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:58:59,841 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:58:59,841 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:58:59,841 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:58:59,841 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,842 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,842 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,842 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:58:59,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:58:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,843 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:58:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:58:59,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:58:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,847 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:58:59,847 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,847 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:58:59,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:58:59,848 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:58:59,848 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:58:59,848 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:58:59,848 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:58:59,948 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:58:59,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:58:59,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:58:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:58:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:58:59,952 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:58:59,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:58:59,952 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:58:59,953 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,953 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:58:59,984 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:58:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:58:59,984 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,984 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:58:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:58:59,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:58:59,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:58:59,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:58:59,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:58:59,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:58:59,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:58:59,985 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:58:59,986 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:58:59,986 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:58:59,986 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:58:59,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:58:59,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:58:59,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:58:59,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:58:59,986 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:00,087 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,087 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:00,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:00,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f108d5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:00,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:00,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,090 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:00,093 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,093 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:00,093 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:00,093 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:00,093 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:00,093 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,094 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:00,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:00,095 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:00,097 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:00,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:00,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:00,099 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:00,100 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:00,099 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:00,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:00,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:00,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/789d31fb592043429de7840cb30ced25 2023-04-01 03:59:00,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9fc6722f0b72427c8d04a7690ea0e03f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f 2023-04-01 03:59:00,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f, entries=150, sequenceid=105, filesize=25.1 K 2023-04-01 03:59:00,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/e28412a411404725b294ee57a86d5641 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641 2023-04-01 03:59:00,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641, entries=150, sequenceid=105, filesize=11.6 K 2023-04-01 03:59:00,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/789d31fb592043429de7840cb30ced25 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25 2023-04-01 03:59:00,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25, entries=150, sequenceid=105, filesize=11.6 K 2023-04-01 03:59:00,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1663ms, sequenceid=105, compaction requested=true 2023-04-01 03:59:00,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:00,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:00,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:00,161 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:00,161 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:00,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:00,162 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:00,162 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103087 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:00,162 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:00,162 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:00,162 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:00,162 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:00,162 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e8f1f56b9bf345aabdd778522c45ebf8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=46.5 K 2023-04-01 03:59:00,162 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/b18e633b9de24abe95c264517b0531bd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=100.7 K 2023-04-01 03:59:00,162 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting b18e633b9de24abe95c264517b0531bd, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c1fdf73b8bf847c8909e55376d8c9778, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680321536132 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9ae9021e9a074042b5cdf0ae65d4b0a8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321537259 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9fc6722f0b72427c8d04a7690ea0e03f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e8f1f56b9bf345aabdd778522c45ebf8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 16227592f8fc4adca4ca53a6cbd9f592, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680321536132 2023-04-01 03:59:00,163 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 6124690a6f064e628e8889c6b0e21c5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321537259 2023-04-01 03:59:00,164 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e28412a411404725b294ee57a86d5641, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:00,172 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:00,176 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#507 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:00,200 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:00,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:00,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:00,206 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:00,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:00,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:00,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:00,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,206 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:00,207 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:00,207 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:00,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,208 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:00,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:00,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:00,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:00,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:00,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:00,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:00,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:00,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:00,209 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:00,209 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:00,209 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:00,210 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:00,210 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:00,583 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/9b2cb7062cfc4ac9b0882a293e5dd699 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/9b2cb7062cfc4ac9b0882a293e5dd699 2023-04-01 03:59:00,586 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 9b2cb7062cfc4ac9b0882a293e5dd699(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:00,586 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:00,586 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=12, startTime=1680321540161; duration=0sec 2023-04-01 03:59:00,586 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:00,586 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:00,587 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:00,587 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:00,587 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:00,588 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/75b0ea63f1b547429e8efa9609eaf370, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=46.5 K 2023-04-01 03:59:00,588 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 75b0ea63f1b547429e8efa9609eaf370, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680321535517 2023-04-01 03:59:00,588 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e951a4a97657488eb8aafb6c6108d169, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680321536132 2023-04-01 03:59:00,588 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4e93d7a06c144b57a3acc97584803a1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680321537259 2023-04-01 03:59:00,588 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 789d31fb592043429de7840cb30ced25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:00,594 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#510 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:00,609 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7373cb945a93497fb740de3fb6a45665 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7373cb945a93497fb740de3fb6a45665 2023-04-01 03:59:00,609 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/ddc3d0797a37455facb8b49546b2b389 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/ddc3d0797a37455facb8b49546b2b389 2023-04-01 03:59:00,619 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 7373cb945a93497fb740de3fb6a45665(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:00,619 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:00,619 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=12, startTime=1680321540161; duration=0sec 2023-04-01 03:59:00,619 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:00,619 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into ddc3d0797a37455facb8b49546b2b389(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:00,619 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:00,619 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=12, startTime=1680321540161; duration=0sec 2023-04-01 03:59:00,619 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:00,627 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401aba5e607cc2f44ac9923255e71515758 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aba5e607cc2f44ac9923255e71515758 2023-04-01 03:59:00,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:00,636 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680321540651, max timestamp=1680321540661Num uniques 859; 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680321540651, max timestamp=1680321540661Num uniques 859; 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680321540651, max timestamp=1680321540661Num uniques 859; 2023-04-01 03:59:00,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 03:59:00,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321600691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321600695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321600696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321600796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321600800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:00,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321600800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321600999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321601004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321601005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,032 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c07d3463a7934669ac3fa39562f94aa0 2023-04-01 03:59:01,043 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/3700e9a3c2954e8db1bc0a0fbfc51cf0 2023-04-01 03:59:01,056 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/7a99476a5b40484fa3cd9fc4c85d0b91 2023-04-01 03:59:01,060 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/c07d3463a7934669ac3fa39562f94aa0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0 2023-04-01 03:59:01,070 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0, entries=150, sequenceid=117, filesize=25.1 K 2023-04-01 03:59:01,071 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/3700e9a3c2954e8db1bc0a0fbfc51cf0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0 2023-04-01 03:59:01,074 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0, entries=150, sequenceid=117, filesize=11.6 K 2023-04-01 03:59:01,075 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/7a99476a5b40484fa3cd9fc4c85d0b91 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91 2023-04-01 03:59:01,078 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91, entries=150, sequenceid=117, filesize=11.6 K 2023-04-01 03:59:01,079 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=129.31 KB/132416 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 869ms, sequenceid=117, compaction requested=false 2023-04-01 03:59:01,079 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:01,079 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:01,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:01,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:01,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:01,079 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:01,080 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:01,080 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:01,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,081 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:01,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5104dc6e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:01,081 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:01,081 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,082 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:01,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:01,082 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:01,082 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,082 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:01,083 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,083 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:01,083 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:01,083 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:01,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:01,084 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:01,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:01,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:01,084 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:01,084 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:01,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,085 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:01,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:01,086 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,087 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:01,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:01,089 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:01,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:01,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:01,089 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,089 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:01,089 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,090 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:01,090 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:01,190 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:01,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:01,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:01,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:01,192 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:01,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:01,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:01,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:01,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:01,195 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:01,195 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:01,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:01,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:01,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:01,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:01,196 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:01,197 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:01,197 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:01,197 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=129.31 KB heapSize=364.54 KB 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:01,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 03:59:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:01,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:01,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321601333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321601334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321601334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321601437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321601437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321601437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321601569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,571 DEBUG [Thread-3649] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:01,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321601576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,578 DEBUG [Thread-3647] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:01,605 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401caa211a2c9194d51ac6353e9679ee3bf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401caa211a2c9194d51ac6353e9679ee3bf 2023-04-01 03:59:01,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321601641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321601642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321601642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321601945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321601945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:01,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321601945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,009 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/2b981ec2c5394452a5c63b3030271d63 2023-04-01 03:59:02,016 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/a7640177b3aa469192e13745c55adef1 2023-04-01 03:59:02,427 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/094de775ff7d4d67804a7cc4951c279a 2023-04-01 03:59:02,430 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/2b981ec2c5394452a5c63b3030271d63 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63 2023-04-01 03:59:02,433 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63, entries=150, sequenceid=155, filesize=25.3 K 2023-04-01 03:59:02,434 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/a7640177b3aa469192e13745c55adef1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1 2023-04-01 03:59:02,437 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1, entries=150, sequenceid=155, filesize=11.8 K 2023-04-01 03:59:02,437 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/094de775ff7d4d67804a7cc4951c279a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a 2023-04-01 03:59:02,440 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a, entries=150, sequenceid=155, filesize=11.8 K 2023-04-01 03:59:02,440 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.31 KB/132416, heapSize ~364.49 KB/373240, currentSize=53.67 KB/54960 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1243ms, sequenceid=155, compaction requested=true 2023-04-01 03:59:02,440 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:02,440 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:02,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:02,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:02,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:02,440 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:02,441 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:02,441 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:02,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,442 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:02,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:02,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39180e1d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:02,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:02,443 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:02,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:02,444 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,444 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:02,444 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:02,444 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:02,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:02,445 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:02,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:02,447 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:02,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:02,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:02,447 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:02,447 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:02,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:02,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,448 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:02,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,449 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:02,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:59:02,450 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:02,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:02,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:02,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:02,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:02,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:02,452 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:02,453 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,452 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:02,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:02,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:02,453 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:02,453 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:02,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:02,454 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:02,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:02,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:02,470 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304017b453f9ab5ba45bb86ed571365300c42 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017b453f9ab5ba45bb86ed571365300c42 2023-04-01 03:59:02,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321602470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321602471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321602473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,487 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/5e0ec02bb9ce4d498f3503cf1431ae55 2023-04-01 03:59:02,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/f679c79c3e16477ba5341526a3775a7c 2023-04-01 03:59:02,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/1387ddad4d564af88e1fff37d6dd1b23 2023-04-01 03:59:02,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/5e0ec02bb9ce4d498f3503cf1431ae55 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55 2023-04-01 03:59:02,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55, entries=150, sequenceid=169, filesize=25.3 K 2023-04-01 03:59:02,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/f679c79c3e16477ba5341526a3775a7c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c 2023-04-01 03:59:02,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c, entries=150, sequenceid=169, filesize=11.8 K 2023-04-01 03:59:02,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/1387ddad4d564af88e1fff37d6dd1b23 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23 2023-04-01 03:59:02,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23, entries=150, sequenceid=169, filesize=11.8 K 2023-04-01 03:59:02,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 85ms, sequenceid=169, compaction requested=true 2023-04-01 03:59:02,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:02,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:02,536 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:02,536 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:02,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:02,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:02,537 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103524 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:02,537 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:02,537 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:02,538 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7373cb945a93497fb740de3fb6a45665, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=101.1 K 2023-04-01 03:59:02,538 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:02,538 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:02,538 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7373cb945a93497fb740de3fb6a45665, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:02,538 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:02,538 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/9b2cb7062cfc4ac9b0882a293e5dd699, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=47.0 K 2023-04-01 03:59:02,538 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting c07d3463a7934669ac3fa39562f94aa0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680321538506 2023-04-01 03:59:02,539 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9b2cb7062cfc4ac9b0882a293e5dd699, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:02,539 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 2b981ec2c5394452a5c63b3030271d63, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321540689 2023-04-01 03:59:02,539 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 3700e9a3c2954e8db1bc0a0fbfc51cf0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680321538506 2023-04-01 03:59:02,539 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5e0ec02bb9ce4d498f3503cf1431ae55, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:02,540 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a7640177b3aa469192e13745c55adef1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321540689 2023-04-01 03:59:02,540 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f679c79c3e16477ba5341526a3775a7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:02,553 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:02,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:02,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:02,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:02,560 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:02,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:02,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:02,560 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:02,560 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:02,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,561 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:02,562 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:02,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,562 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:02,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:02,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:02,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:02,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:02,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:02,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:02,568 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:02,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:02,570 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:02,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:02,570 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:02,570 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:02,571 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:02,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:02,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:02,591 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#520 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:02,597 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/d4e6e5ce17664d1db299ac2212e99169 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d4e6e5ce17664d1db299ac2212e99169 2023-04-01 03:59:02,603 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into d4e6e5ce17664d1db299ac2212e99169(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:02,603 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:02,603 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=12, startTime=1680321542536; duration=0sec 2023-04-01 03:59:02,603 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:02,603 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 03:59:02,605 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 03:59:02,605 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:02,605 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:02,605 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/ddc3d0797a37455facb8b49546b2b389, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=47.0 K 2023-04-01 03:59:02,606 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting ddc3d0797a37455facb8b49546b2b389, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680321538388 2023-04-01 03:59:02,607 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7a99476a5b40484fa3cd9fc4c85d0b91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680321538506 2023-04-01 03:59:02,608 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 094de775ff7d4d67804a7cc4951c279a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680321540689 2023-04-01 03:59:02,609 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1387ddad4d564af88e1fff37d6dd1b23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:02,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321602604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321602604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321602607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,657 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:02,677 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401a8350a1177614b8d8905660fec67e00b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a8350a1177614b8d8905660fec67e00b 2023-04-01 03:59:02,709 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/e4f34803826b478b88cd7ad32e5c8203 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e4f34803826b478b88cd7ad32e5c8203 2023-04-01 03:59:02,713 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into e4f34803826b478b88cd7ad32e5c8203(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:02,713 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:02,713 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=12, startTime=1680321542536; duration=0sec 2023-04-01 03:59:02,713 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:02,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321602713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321602714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321602714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321602917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321602917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:02,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:02,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40564 deadline: 1680321602919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,116 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9b8bc49083fb4463be86d099d9366c00 2023-04-01 03:59:03,158 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/0fab7bd5d6ba448398b23046b3b20eba as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0fab7bd5d6ba448398b23046b3b20eba 2023-04-01 03:59:03,163 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 0fab7bd5d6ba448398b23046b3b20eba(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:03,163 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:03,163 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=12, startTime=1680321542536; duration=0sec 2023-04-01 03:59:03,163 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:03,168 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/703de9660d68471481fbdd4b770cd590 2023-04-01 03:59:03,176 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/b2559681e9de4d87860d1ef41a44faa4 2023-04-01 03:59:03,181 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9b8bc49083fb4463be86d099d9366c00 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00 2023-04-01 03:59:03,196 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00, entries=150, sequenceid=187, filesize=25.3 K 2023-04-01 03:59:03,197 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/703de9660d68471481fbdd4b770cd590 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590 2023-04-01 03:59:03,220 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590, entries=150, sequenceid=187, filesize=11.8 K 2023-04-01 03:59:03,221 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/b2559681e9de4d87860d1ef41a44faa4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4 2023-04-01 03:59:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321603221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321603222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,225 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4, entries=150, sequenceid=187, filesize=11.8 K 2023-04-01 03:59:03,226 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 656ms, sequenceid=187, compaction requested=false 2023-04-01 03:59:03,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:03,227 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,227 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,229 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,229 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:03,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:59:03,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:03,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:03,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:03,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:03,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5134cde4[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,231 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,231 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,232 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,232 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,256 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,256 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,259 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,259 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,260 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,260 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,261 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,262 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,267 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,269 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401840fa3c7f50f45729daf1c16bc052837 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401840fa3c7f50f45729daf1c16bc052837 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,283 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,284 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,284 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,283 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,284 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,284 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,284 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321603283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,384 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:03,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:03,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321603387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:03,389 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:03,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:03,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:03,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,389 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,391 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,391 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:03,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:03,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:03,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:03,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:03,392 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:03,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:03,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:03,393 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,394 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,394 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c3d0b7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,415 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,415 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,416 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,416 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,418 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,418 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,418 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,418 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,418 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,419 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,419 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,419 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,420 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,421 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,424 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,424 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,424 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,424 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,524 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:03,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:03,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:03,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:03,528 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:03,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:03,528 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:03,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,529 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,529 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:03,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:03,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:03,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:03,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:03,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,534 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:03,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:03,538 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:03,538 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,538 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:03,539 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:03,539 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,540 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d3994d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,544 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,545 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,548 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,548 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,548 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,558 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,559 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,559 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:03,559 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,559 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321603589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,659 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:03,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:03,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:03,662 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:03,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:03,662 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:03,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,663 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,664 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:03,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:03,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:03,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:03,665 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:03,665 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:03,665 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:03,665 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,665 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,666 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,666 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f420b6c[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,667 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,667 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,668 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,668 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,668 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,668 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,669 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,669 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,672 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,672 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,672 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,674 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,675 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,676 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,678 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,678 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,678 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,692 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=205, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/5cd345d9c2df42b7a8a2a5c45419b92b 2023-04-01 03:59:03,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321603725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321603726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,779 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:03,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:03,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:03,781 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:03,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:03,781 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:03,782 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,782 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,784 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,784 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:03,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:03,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:03,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:03,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:03,785 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,785 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,787 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,787 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,787 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,787 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68eebe8d[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,788 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,788 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,789 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,789 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,789 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,790 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,791 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,791 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,791 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,791 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,793 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,793 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,793 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,796 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,798 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,799 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,799 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,799 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:03,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321603892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,899 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:03,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:03,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:03,902 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:03,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:03,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:03,902 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,903 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,906 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,906 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:03,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:03,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:03,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,907 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:03,907 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:03,908 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:03,908 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:03,908 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,908 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,909 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,909 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:03,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a78d4f[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:03,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:03,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,910 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:03,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:03,910 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:03,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,910 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:03,911 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,911 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:03,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:03,913 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:03,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:03,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:03,913 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:03,913 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:03,914 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:03,914 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:03,914 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,915 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,916 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:03,916 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:03,916 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:03,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:03,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:03,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:03,916 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:03,916 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,917 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:03,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:03,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:03,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:03,917 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:03,917 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:03,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:04,017 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:04,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:04,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:04,020 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,021 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,021 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:04,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:04,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:04,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:04,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:04,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:04,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:04,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:04,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:04,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,024 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:04,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc47907[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:04,025 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:04,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,026 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,026 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:04,026 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:04,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:04,026 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:04,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:04,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:04,027 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,027 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:04,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:04,029 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:04,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:04,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:04,029 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:04,029 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:04,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:04,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:04,030 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,031 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,032 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:04,032 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:04,032 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:04,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:04,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:04,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:04,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:04,032 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,033 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:04,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,033 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:04,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:04,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:04,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/228f659a5c1749f490054e5fb557f9f4 2023-04-01 03:59:04,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/93598f02d8ad44abad2b828cb038d8e9 2023-04-01 03:59:04,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/5cd345d9c2df42b7a8a2a5c45419b92b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b 2023-04-01 03:59:04,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b, entries=200, sequenceid=205, filesize=32.1 K 2023-04-01 03:59:04,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/228f659a5c1749f490054e5fb557f9f4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4 2023-04-01 03:59:04,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4, entries=150, sequenceid=205, filesize=11.8 K 2023-04-01 03:59:04,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/93598f02d8ad44abad2b828cb038d8e9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9 2023-04-01 03:59:04,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9, entries=150, sequenceid=205, filesize=11.8 K 2023-04-01 03:59:04,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 894ms, sequenceid=205, compaction requested=true 2023-04-01 03:59:04,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:04,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:04,124 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:04,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:04,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:04,124 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:04,125 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:04,125 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:04,125 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:04,125 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:04,125 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,125 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,125 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0fab7bd5d6ba448398b23046b3b20eba, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=83.1 K 2023-04-01 03:59:04,125 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d4e6e5ce17664d1db299ac2212e99169, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=35.6 K 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0fab7bd5d6ba448398b23046b3b20eba, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d4e6e5ce17664d1db299ac2212e99169, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 9b8bc49083fb4463be86d099d9366c00, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680321542465 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 703de9660d68471481fbdd4b770cd590, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680321542465 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 5cd345d9c2df42b7a8a2a5c45419b92b, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:04,126 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 228f659a5c1749f490054e5fb557f9f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:04,130 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#529 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:04,133 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:04,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:04,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:04,140 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#528 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:04,142 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:04,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:04,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:04,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:04,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:04,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:04,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:04,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:04,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:04,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:04,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:04,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:04,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:04,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:04,146 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:04,146 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,146 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:04,146 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:04,147 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:04,153 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/0e060f47ba0b4e2ead1ad7581cd2c0d1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0e060f47ba0b4e2ead1ad7581cd2c0d1 2023-04-01 03:59:04,156 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 0e060f47ba0b4e2ead1ad7581cd2c0d1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:04,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:04,156 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=13, startTime=1680321544124; duration=0sec 2023-04-01 03:59:04,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:04,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:04,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:04,156 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:04,157 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:04,157 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e4f34803826b478b88cd7ad32e5c8203, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=35.6 K 2023-04-01 03:59:04,157 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting e4f34803826b478b88cd7ad32e5c8203, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680321542449 2023-04-01 03:59:04,157 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting b2559681e9de4d87860d1ef41a44faa4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680321542465 2023-04-01 03:59:04,157 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 93598f02d8ad44abad2b828cb038d8e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:04,168 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:04,173 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401e4d0d182e6944a4e832d853aea8ed486 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e4d0d182e6944a4e832d853aea8ed486 2023-04-01 03:59:04,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:04,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:04,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321604424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:04,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321604526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,582 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/9fe99134d6884ca68c2632ad709a6fc3 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/9fe99134d6884ca68c2632ad709a6fc3 2023-04-01 03:59:04,585 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 9fe99134d6884ca68c2632ad709a6fc3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:04,586 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:04,586 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=13, startTime=1680321544124; duration=0sec 2023-04-01 03:59:04,586 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:04,590 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=222, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/410fb013b5c8466f88656d9b859b650e 2023-04-01 03:59:04,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:04,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321604728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:04,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321604736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:04,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321604736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:04,968 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/41cc56d4905f433ca5821aedbf9d05a9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/41cc56d4905f433ca5821aedbf9d05a9 2023-04-01 03:59:04,971 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 41cc56d4905f433ca5821aedbf9d05a9(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:04,971 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:04,971 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=13, startTime=1680321544124; duration=0sec 2023-04-01 03:59:04,971 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:04,998 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/eb72ba60b0ed4376bb180f87f1e6016b 2023-04-01 03:59:05,010 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/03ad7a6bd32142eb834daf69a18f5684 2023-04-01 03:59:05,013 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/410fb013b5c8466f88656d9b859b650e as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e 2023-04-01 03:59:05,016 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e, entries=150, sequenceid=222, filesize=25.3 K 2023-04-01 03:59:05,016 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/eb72ba60b0ed4376bb180f87f1e6016b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b 2023-04-01 03:59:05,019 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b, entries=150, sequenceid=222, filesize=11.8 K 2023-04-01 03:59:05,020 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/03ad7a6bd32142eb834daf69a18f5684 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684 2023-04-01 03:59:05,022 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684, entries=150, sequenceid=222, filesize=11.8 K 2023-04-01 03:59:05,023 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 877ms, sequenceid=222, compaction requested=false 2023-04-01 03:59:05,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:05,023 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,023 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,024 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,024 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,025 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12d218b8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,026 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,028 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,028 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,028 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,028 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,028 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,028 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,030 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,032 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,032 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,032 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:05,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:59:05,032 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:05,032 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:05,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:05,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:05,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:05,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:05,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,033 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,034 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,036 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,039 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,039 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,039 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,039 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:05,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321605062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,139 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,141 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,142 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,142 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,143 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,144 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,144 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,144 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,144 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f55c6b9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,146 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,147 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,148 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,148 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,150 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,150 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,150 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,150 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,150 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,151 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,152 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,153 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,153 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,154 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,154 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,154 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,154 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,154 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:05,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321605164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,254 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,257 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,257 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,259 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,260 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,260 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,260 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,261 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,261 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,261 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,262 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,263 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,263 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5443ffe9[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,264 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,264 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,265 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,265 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,265 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,266 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,266 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,268 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,268 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,268 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,270 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,271 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,274 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,274 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,274 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,274 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:05,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321605366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,374 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,382 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,383 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,384 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,384 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,385 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,389 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,389 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,389 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,390 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,390 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,390 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,390 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,390 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,390 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,390 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,390 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,392 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,392 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6af1a827[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,394 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,394 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,395 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,396 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,397 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,404 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,404 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,405 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,405 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,406 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,406 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,407 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,408 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,410 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,414 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,414 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,415 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,415 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,415 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,442 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040135878260ca5c4bea833aaa510cb6379c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040135878260ca5c4bea833aaa510cb6379c 2023-04-01 03:59:05,515 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,522 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,522 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,524 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,524 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,525 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,526 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,526 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,528 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,529 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e46a19[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,530 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,532 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,532 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,532 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,536 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,536 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,538 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,538 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,538 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,539 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,539 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,539 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,540 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,541 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,542 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,543 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,543 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,643 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,646 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,646 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,663 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,663 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,664 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,664 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,664 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,664 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,667 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,667 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492509b2[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,668 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:05,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321605670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,690 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,691 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,691 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,692 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,693 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,693 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,693 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,694 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,695 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,696 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,696 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,696 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,696 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,796 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,799 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,799 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,800 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,845 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,846 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,850 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,851 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,851 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,852 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,853 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7cd6570db3b54f6a883df066ada44ee9 2023-04-01 03:59:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2521bc8[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,853 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,855 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,855 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,856 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,856 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,857 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,860 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,860 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,860 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,862 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,862 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,862 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,862 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,865 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,875 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,875 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,875 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,875 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,976 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:05,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:05,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:05,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:05,978 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:05,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:05,978 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:05,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,979 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,981 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:05,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:05,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:05,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:05,982 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:05,983 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:05,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:05,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:05,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:05,983 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:05,984 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:05,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65053b35[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:05,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:05,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,987 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:05,987 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,988 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:05,988 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:05,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:05,991 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:05,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:05,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:05,991 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:05,991 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:05,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,991 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:05,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:05,992 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,993 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:05,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:05,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:05,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:05,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:05,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:05,995 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:05,995 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:05,995 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:05,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:05,995 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:05,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,002 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,096 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,098 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,098 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,099 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,100 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,100 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,101 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,101 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,101 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,101 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,102 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2efa312e[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,103 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,104 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,104 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,104 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,105 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,107 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,107 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,107 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,107 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,107 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,108 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,108 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,109 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,111 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:06,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,111 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,111 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,111 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:06,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321606174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,211 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,214 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,214 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,214 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,214 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,214 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,214 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,215 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,215 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,215 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,215 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,216 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,216 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,216 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,216 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,216 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,220 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,221 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f73d5da[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,223 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,223 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,224 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,224 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,224 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,226 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,229 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,229 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,229 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,231 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,232 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,234 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,238 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,238 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,238 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,239 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/747aed9faaab4ff6ae0ae47e2fed17fd 2023-04-01 03:59:06,339 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,342 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,343 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,345 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,345 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,347 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,348 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6d0ba7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,350 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,351 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,351 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,352 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,352 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,355 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,356 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,358 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,359 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,359 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,359 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:06,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,459 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,461 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,461 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,462 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,463 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,463 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,464 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,464 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,465 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,465 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,465 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,466 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,466 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,466 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,466 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117d5fac[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,467 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,467 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,468 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,467 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,468 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,468 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,468 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,468 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,471 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,471 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,471 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,472 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,473 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,474 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,474 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,475 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,475 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,475 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,575 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,578 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,578 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,580 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,580 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,590 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,590 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,591 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,592 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,592 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,593 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e716fec[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,594 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,596 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,596 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,597 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,598 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,598 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,600 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,600 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,600 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,601 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,601 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,602 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,605 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:06,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,605 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,606 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,606 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,706 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,709 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,709 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,709 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,709 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,709 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,709 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,711 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,711 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,714 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:06,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:06,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:06,716 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:06,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7dbafd[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:06,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:06,718 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,720 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:06,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:06,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,720 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:06,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:06,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:06,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:06,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:06,724 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:06,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:06,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:06,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:06,724 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:06,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4fa05c23fe574680bad94c815a03e959 2023-04-01 03:59:06,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,725 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,726 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,727 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7cd6570db3b54f6a883df066ada44ee9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,729 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:06,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:06,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,730 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:06,730 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:06,730 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:06,730 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:06,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:06,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9, entries=200, sequenceid=241, filesize=32.1 K 2023-04-01 03:59:06,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/747aed9faaab4ff6ae0ae47e2fed17fd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd 2023-04-01 03:59:06,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd, entries=150, sequenceid=241, filesize=11.8 K 2023-04-01 03:59:06,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4fa05c23fe574680bad94c815a03e959 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959 2023-04-01 03:59:06,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:06,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321606742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:06,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321606743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,745 DEBUG [Thread-3645] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:06,746 DEBUG [Thread-3643] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:06,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959, entries=150, sequenceid=241, filesize=11.8 K 2023-04-01 03:59:06,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1716ms, sequenceid=241, compaction requested=true 2023-04-01 03:59:06,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:06,748 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:06,748 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:06,748 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:06,748 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:06,748 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:06,749 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:06,749 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:06,749 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,749 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0e060f47ba0b4e2ead1ad7581cd2c0d1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=35.7 K 2023-04-01 03:59:06,750 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85156 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:06,750 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:06,750 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,750 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/41cc56d4905f433ca5821aedbf9d05a9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=83.2 K 2023-04-01 03:59:06,750 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 0e060f47ba0b4e2ead1ad7581cd2c0d1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:06,750 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 41cc56d4905f433ca5821aedbf9d05a9, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:06,751 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting eb72ba60b0ed4376bb180f87f1e6016b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680321543267 2023-04-01 03:59:06,751 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 410fb013b5c8466f88656d9b859b650e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680321543267 2023-04-01 03:59:06,751 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 747aed9faaab4ff6ae0ae47e2fed17fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:06,751 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 7cd6570db3b54f6a883df066ada44ee9, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:06,761 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#537 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:06,782 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#538 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:06,830 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:06,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:06,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:06,832 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:06,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:06,832 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:06,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,833 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,837 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:06,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:06,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:06,837 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:06,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:06,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:06,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:06,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:06,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:06,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:06,838 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:06,839 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:06,839 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:06,839 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:06,850 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401adfd589bfdc947fcb30a717d1c709050 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401adfd589bfdc947fcb30a717d1c709050 2023-04-01 03:59:06,853 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/6182ba014eb74f9baafd9905f0b5d55c 2023-04-01 03:59:07,179 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/eb02ee12c36543e690ac89657eaef1a2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb02ee12c36543e690ac89657eaef1a2 2023-04-01 03:59:07,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:07,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:07,185 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into eb02ee12c36543e690ac89657eaef1a2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:07,185 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:07,185 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=13, startTime=1680321546748; duration=0sec 2023-04-01 03:59:07,185 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:07,185 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:07,187 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:07,189 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:07,189 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:07,189 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/9fe99134d6884ca68c2632ad709a6fc3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=35.7 K 2023-04-01 03:59:07,189 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 9fe99134d6884ca68c2632ad709a6fc3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680321542601 2023-04-01 03:59:07,190 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 03ad7a6bd32142eb834daf69a18f5684, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680321543267 2023-04-01 03:59:07,190 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 4fa05c23fe574680bad94c815a03e959, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:07,195 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#541 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:07,218 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/a60f759c73634b2a8169870e64dc6eb1 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/a60f759c73634b2a8169870e64dc6eb1 2023-04-01 03:59:07,222 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into a60f759c73634b2a8169870e64dc6eb1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:07,222 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:07,222 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=13, startTime=1680321546748; duration=0sec 2023-04-01 03:59:07,222 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:07,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:07,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321607224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,260 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/cd44a504d9414517b1d9cdc580f469e9 2023-04-01 03:59:07,325 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/653977957503447b85fbbdab3cfbbb87 2023-04-01 03:59:07,329 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/6182ba014eb74f9baafd9905f0b5d55c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c 2023-04-01 03:59:07,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:07,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321607332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,335 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c, entries=150, sequenceid=257, filesize=25.3 K 2023-04-01 03:59:07,336 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/cd44a504d9414517b1d9cdc580f469e9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9 2023-04-01 03:59:07,340 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9, entries=150, sequenceid=257, filesize=11.8 K 2023-04-01 03:59:07,341 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/653977957503447b85fbbdab3cfbbb87 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87 2023-04-01 03:59:07,343 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87, entries=150, sequenceid=257, filesize=11.8 K 2023-04-01 03:59:07,344 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 505ms, sequenceid=257, compaction requested=false 2023-04-01 03:59:07,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:07,344 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:07,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:07,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:07,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:07,344 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:07,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:07,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:07,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,347 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@af8ae86[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:07,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:07,349 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,349 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:07,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:07,350 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:07,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,350 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:07,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:07,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:07,351 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:07,351 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:07,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:07,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:07,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:07,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:07,354 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:07,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:07,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,356 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:07,356 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,358 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:07,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,375 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:07,375 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:07,475 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:07,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:07,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:07,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:07,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:07,480 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:07,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:07,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:07,480 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:07,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,481 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:07,485 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:07,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:07,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,485 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:07,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:07,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:07,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:07,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:07,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:07,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:07,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:07,486 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:07,486 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:07,487 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:07,487 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:07,487 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:07,498 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304019c2c502385174913b46f4f102f77fa67 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019c2c502385174913b46f4f102f77fa67 2023-04-01 03:59:07,506 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/98a297878a564de19a1fed96027f5514 2023-04-01 03:59:07,523 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/135ea3240427437ab6d3177039fa5dd2 2023-04-01 03:59:07,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:07,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:07,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:07,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321607584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:07,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321607688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:07,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321607893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,956 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/2ec031c8c6524ee698e43a8c80e6af69 2023-04-01 03:59:07,959 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/98a297878a564de19a1fed96027f5514 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514 2023-04-01 03:59:07,962 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514, entries=150, sequenceid=275, filesize=25.4 K 2023-04-01 03:59:07,962 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/135ea3240427437ab6d3177039fa5dd2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2 2023-04-01 03:59:07,966 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2, entries=150, sequenceid=275, filesize=11.9 K 2023-04-01 03:59:07,966 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/2ec031c8c6524ee698e43a8c80e6af69 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69 2023-04-01 03:59:07,975 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69, entries=150, sequenceid=275, filesize=11.9 K 2023-04-01 03:59:07,976 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 489ms, sequenceid=275, compaction requested=true 2023-04-01 03:59:07,976 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:07,976 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:07,978 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:07,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:07,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7183104a[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:07,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:07,980 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,980 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:07,981 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:07,981 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:07,981 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:07,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,981 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:07,982 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:07,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:07,982 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:07,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:07,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:07,985 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:07,985 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:07,985 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:07,985 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:07,985 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:08,006 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,007 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:08,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,007 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:08,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,008 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:08,009 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:08,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:08,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:08,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,012 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:08,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:08,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:08,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:08,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:08,013 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:08,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:08,013 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:08,013 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:08,013 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:08,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:08,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:08,014 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/e346f9cd864645179620f499800af6e6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/e346f9cd864645179620f499800af6e6 2023-04-01 03:59:08,019 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into e346f9cd864645179620f499800af6e6(size=25.9 K), total size for store is 76.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:59:08,019 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:08,019 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=13, startTime=1680321546748; duration=1sec 2023-04-01 03:59:08,019 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:08,113 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:08,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:08,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:08,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:08,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:08,115 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:08,115 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:08,115 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:08,115 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:08,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,116 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:08,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:08,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:08,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,118 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:08,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:08,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:08,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:08,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:08,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:08,119 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:08,119 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:08,119 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:08,119 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:08,119 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:08,119 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:08,119 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:08,120 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:08,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:08,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:08,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:08,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321608237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:08,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321608340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,530 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018f01a1d3dd424739acc8237cd0b9871d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018f01a1d3dd424739acc8237cd0b9871d 2023-04-01 03:59:08,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:08,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321608543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321608845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:08,934 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=292, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/89d4b2a9a5fe4f4487c940c45bf00599 2023-04-01 03:59:09,342 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/c777d448f133465194eedf6a449af919 2023-04-01 03:59:09,349 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/dde6ff05ce734554bb33fd3bc0a9eb0d 2023-04-01 03:59:09,352 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/89d4b2a9a5fe4f4487c940c45bf00599 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599 2023-04-01 03:59:09,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321609350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,354 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599, entries=150, sequenceid=292, filesize=25.4 K 2023-04-01 03:59:09,354 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/c777d448f133465194eedf6a449af919 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919 2023-04-01 03:59:09,357 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919, entries=150, sequenceid=292, filesize=11.9 K 2023-04-01 03:59:09,358 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/dde6ff05ce734554bb33fd3bc0a9eb0d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d 2023-04-01 03:59:09,361 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d, entries=150, sequenceid=292, filesize=11.9 K 2023-04-01 03:59:09,361 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1242ms, sequenceid=292, compaction requested=true 2023-04-01 03:59:09,361 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:09,361 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,363 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:09,369 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14680f99[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:09,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:09,371 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,375 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:09,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,375 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:09,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:09,375 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:09,376 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:09,378 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:09,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:09,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:09,378 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:09,378 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:09,379 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,379 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:09,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,379 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:09,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:09,379 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,381 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,382 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:09,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:09,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:09,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:09,382 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,383 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,383 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:09,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:09,383 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:09,383 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:09,383 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:09,483 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:09,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:09,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:09,486 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:09,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:09,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:09,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:09,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,486 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:09,490 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:09,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:09,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:09,490 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:09,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:09,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:09,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:09,491 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:09,491 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:09,492 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:09,518 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401fb17a4b80456423a866f66a3ec98be93 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fb17a4b80456423a866f66a3ec98be93 2023-04-01 03:59:09,521 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/0578f1ca749b4a9eac75c6279351fd36 2023-04-01 03:59:09,942 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/d89cc3019277418eb143f3a3caecea23 2023-04-01 03:59:10,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:10,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:10,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321610400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,412 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/bec02507afd9456d94fd5836aca3b1fc 2023-04-01 03:59:10,418 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/0578f1ca749b4a9eac75c6279351fd36 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36 2023-04-01 03:59:10,421 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36, entries=150, sequenceid=308, filesize=25.4 K 2023-04-01 03:59:10,422 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/d89cc3019277418eb143f3a3caecea23 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23 2023-04-01 03:59:10,424 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23, entries=150, sequenceid=308, filesize=11.9 K 2023-04-01 03:59:10,425 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/bec02507afd9456d94fd5836aca3b1fc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc 2023-04-01 03:59:10,427 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc, entries=150, sequenceid=308, filesize=11.9 K 2023-04-01 03:59:10,428 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 937ms, sequenceid=308, compaction requested=true 2023-04-01 03:59:10,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:10,428 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:10,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,430 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,430 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:10,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4690f5bb[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:10,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:10,431 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:10,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,432 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:10,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:10,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:10,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:10,432 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,432 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,433 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:10,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:10,435 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:10,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:10,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:10,435 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:10,435 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:10,436 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,436 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,436 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,436 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,437 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:10,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:10,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,439 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,439 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,439 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:10,439 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:10,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:10,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:10,515 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401055647ff87474ec9a09bdf9f5d576b01 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401055647ff87474ec9a09bdf9f5d576b01 2023-04-01 03:59:10,539 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:10,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:10,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:10,542 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:10,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:10,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:10,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,542 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,543 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,543 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,544 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:10,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:10,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:10,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:10,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,545 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:10,546 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,546 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:10,546 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:10,546 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:10,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321610544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,546 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:10,547 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:10,547 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,547 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,548 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,548 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:10,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d3ef71[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:10,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:10,549 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,550 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:10,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:10,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,550 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:10,550 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:10,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:10,551 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,551 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:10,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:10,552 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:10,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:10,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:10,552 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:10,552 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:10,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,553 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,554 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,556 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,574 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,575 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:10,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,575 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,575 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,575 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321610647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,675 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:10,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:10,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:10,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:10,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:10,678 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:10,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:10,678 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:10,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,679 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,680 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,680 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,681 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:10,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:10,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:10,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:10,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,681 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:10,682 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,682 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:10,682 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:10,682 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:10,682 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:10,684 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,684 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,684 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,684 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:10,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603bbf41[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:10,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:10,685 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,686 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:10,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:10,686 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:10,686 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,686 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:10,687 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,687 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:10,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:10,688 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:10,688 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:10,688 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:10,688 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:10,689 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:10,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,689 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,690 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,692 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:10,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:10,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:10,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,692 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:10,692 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,692 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1680321610758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,763 DEBUG [Thread-3643] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:10,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40552 deadline: 1680321610780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,784 DEBUG [Thread-3645] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., hostname=jenkins-hbase20.apache.org,36251,1680321375816, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 03:59:10,792 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:10,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:10,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:10,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:10,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:10,795 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:10,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:10,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:10,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,795 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,801 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,801 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:10,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:10,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:10,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:10,802 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:10,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:10,803 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:10,803 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,804 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:10,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@347b6018[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:10,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:10,805 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,806 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:10,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,806 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:10,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:10,806 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:10,807 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:10,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,807 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:10,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:10,809 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:10,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:10,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:10,809 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:10,809 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:10,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,810 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,810 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,811 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:10,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,813 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:10,812 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,812 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,813 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,813 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,813 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:10,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,813 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:10,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:10,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321610852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,913 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:10,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:10,916 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:10,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:10,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:10,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,917 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,918 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,918 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,919 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,919 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=325, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/431502e6401544529288c67e5c22acf0 2023-04-01 03:59:10,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:10,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:10,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:10,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:10,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:10,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:10,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:10,920 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:10,921 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,921 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:10,921 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:10,921 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:10,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:10,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:10,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:10,921 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:10,923 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,924 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:10,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75877601[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:10,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:10,925 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,930 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,931 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:10,931 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:10,931 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:10,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,931 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:10,933 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,933 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:10,933 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,933 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:10,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:10,936 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:10,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:10,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:10,936 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:10,936 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:10,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,938 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:10,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,939 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:10,940 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:10,944 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:10,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:10,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:10,944 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:10,944 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:10,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:10,944 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,044 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:11,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:11,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:11,047 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:11,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:11,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:11,047 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,048 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,049 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,049 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,050 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:11,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:11,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:11,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:11,051 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:11,051 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:11,051 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:11,051 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:11,051 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:11,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:11,052 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:11,053 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,053 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:11,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6339a7a7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:11,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:11,058 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,060 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:11,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:11,060 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:11,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,060 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:11,061 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:11,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:11,064 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:11,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:11,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:11,064 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:11,065 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:11,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,068 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,068 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,070 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,072 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:11,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:11,072 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,073 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,073 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:11,073 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:11,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:11,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321611155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,173 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:11,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:11,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:11,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:11,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:11,175 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:11,175 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:11,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:11,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,176 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,177 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,177 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,178 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:11,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:11,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:11,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:11,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:11,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:11,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:11,182 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:11,183 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,183 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:11,183 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,185 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:11,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e45cd36[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:11,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:11,186 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,188 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:11,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:11,188 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,188 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:11,188 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:11,190 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,190 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:11,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:11,192 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:11,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:11,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:11,192 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:11,192 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:11,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,196 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,197 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,198 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:11,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:11,198 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,199 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:11,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,199 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:11,199 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,299 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:11,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:11,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:11,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:11,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:11,302 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:11,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:11,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:11,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,302 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,317 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,317 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:11,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:11,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:11,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:11,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:11,318 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:11,319 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:11,325 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,325 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:11,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d11d46[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:11,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:11,326 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/29a27216375f47b4aa43f1807966cbee 2023-04-01 03:59:11,342 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:11,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,342 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:11,343 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:11,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/1f35eec1278e42e7a20db08cfd027ff5 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,343 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:11,345 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:11,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:11,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:11,345 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:11,345 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:11,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/431502e6401544529288c67e5c22acf0 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0 2023-04-01 03:59:11,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,346 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:11,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,347 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,348 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:11,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0, entries=200, sequenceid=325, filesize=32.3 K 2023-04-01 03:59:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/29a27216375f47b4aa43f1807966cbee as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee 2023-04-01 03:59:11,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee, entries=150, sequenceid=325, filesize=11.9 K 2023-04-01 03:59:11,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/1f35eec1278e42e7a20db08cfd027ff5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,354 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:11,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:11,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,354 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:11,354 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:11,355 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,354 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:11,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:11,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5, entries=150, sequenceid=325, filesize=11.9 K 2023-04-01 03:59:11,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 857ms, sequenceid=325, compaction requested=true 2023-04-01 03:59:11,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:11,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:11,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:11,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:11,362 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:59:11,362 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:59:11,364 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163596 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:59:11,364 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:11,364 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,364 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/e346f9cd864645179620f499800af6e6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=159.8 K 2023-04-01 03:59:11,364 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting e346f9cd864645179620f499800af6e6, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73396 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:11,365 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6182ba014eb74f9baafd9905f0b5d55c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680321545057 2023-04-01 03:59:11,365 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb02ee12c36543e690ac89657eaef1a2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=71.7 K 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 98a297878a564de19a1fed96027f5514, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680321547215 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting eb02ee12c36543e690ac89657eaef1a2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 89d4b2a9a5fe4f4487c940c45bf00599, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680321547573 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting cd44a504d9414517b1d9cdc580f469e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680321545057 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 0578f1ca749b4a9eac75c6279351fd36, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1680321548223 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 135ea3240427437ab6d3177039fa5dd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680321547215 2023-04-01 03:59:11,365 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 431502e6401544529288c67e5c22acf0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:11,366 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting c777d448f133465194eedf6a449af919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680321547573 2023-04-01 03:59:11,366 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting d89cc3019277418eb143f3a3caecea23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1680321548223 2023-04-01 03:59:11,366 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 29a27216375f47b4aa43f1807966cbee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:11,376 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#555 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:11,378 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#556 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:11,382 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/7ca5c726462c482d86d643591e2b5192 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/7ca5c726462c482d86d643591e2b5192 2023-04-01 03:59:11,387 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 7ca5c726462c482d86d643591e2b5192(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:11,387 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:11,387 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=10, startTime=1680321551362; duration=0sec 2023-04-01 03:59:11,388 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:11,388 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 03:59:11,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73396 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 03:59:11,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:11,389 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,389 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/a60f759c73634b2a8169870e64dc6eb1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=71.7 K 2023-04-01 03:59:11,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting a60f759c73634b2a8169870e64dc6eb1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680321544407 2023-04-01 03:59:11,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 653977957503447b85fbbdab3cfbbb87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680321545057 2023-04-01 03:59:11,389 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2ec031c8c6524ee698e43a8c80e6af69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680321547215 2023-04-01 03:59:11,390 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting dde6ff05ce734554bb33fd3bc0a9eb0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680321547573 2023-04-01 03:59:11,390 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting bec02507afd9456d94fd5836aca3b1fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1680321548223 2023-04-01 03:59:11,390 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 1f35eec1278e42e7a20db08cfd027ff5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:11,395 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#557 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:11,455 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:11,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:11,457 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:11,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:11,457 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:11,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,458 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,460 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:11,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,460 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:11,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:11,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:11,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:11,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:11,461 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:11,461 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:11,461 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 03:59:11,461 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:11,462 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:11,462 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:11,462 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:11,462 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:11,462 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:11,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:11,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:11,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321611650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321611651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321611658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321611754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321611754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,801 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/aab836daf4fd4cc6a72942e8c4bc58f9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aab836daf4fd4cc6a72942e8c4bc58f9 2023-04-01 03:59:11,805 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into aab836daf4fd4cc6a72942e8c4bc58f9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:11,805 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:11,805 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=10, startTime=1680321551362; duration=0sec 2023-04-01 03:59:11,805 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:11,871 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401f4666d44a7af4c7d9242b2f6e591f315 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f4666d44a7af4c7d9242b2f6e591f315 2023-04-01 03:59:11,886 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/21dcbffea2da4c2f8424439e9c18511f 2023-04-01 03:59:11,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321611956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:11,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321611957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,195 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/6d8ed9674b2a4016ae3b52b5bf4de63b as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6d8ed9674b2a4016ae3b52b5bf4de63b 2023-04-01 03:59:12,198 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 6d8ed9674b2a4016ae3b52b5bf4de63b(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:12,198 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:12,198 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=10, startTime=1680321551362; duration=0sec 2023-04-01 03:59:12,198 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:12,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321612258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321612258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,293 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2872351d637e449b89bd31f7f0e380e9 2023-04-01 03:59:12,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40564 deadline: 1680321612662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,701 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/f93bba7e749042c2b464c6aeed86e3f2 2023-04-01 03:59:12,704 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/21dcbffea2da4c2f8424439e9c18511f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f 2023-04-01 03:59:12,708 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f, entries=150, sequenceid=341, filesize=25.4 K 2023-04-01 03:59:12,708 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2872351d637e449b89bd31f7f0e380e9 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9 2023-04-01 03:59:12,711 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9, entries=150, sequenceid=341, filesize=11.9 K 2023-04-01 03:59:12,712 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/f93bba7e749042c2b464c6aeed86e3f2 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2 2023-04-01 03:59:12,715 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2, entries=150, sequenceid=341, filesize=11.9 K 2023-04-01 03:59:12,715 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 1254ms, sequenceid=341, compaction requested=false 2023-04-01 03:59:12,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:12,715 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:12,715 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:12,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:12,716 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,719 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:12,720 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:12,722 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,722 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,723 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,723 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:12,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f284203[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:12,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:12,724 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,728 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:12,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,728 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:12,729 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:12,730 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:12,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:12,732 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:12,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:12,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:12,732 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:12,732 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:12,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:12,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,733 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:12,733 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,734 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:12,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:12,737 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:12,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:12,737 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,737 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:12,737 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:12,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:12,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:12,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:12,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36251] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321612774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321612775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,837 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:12,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:12,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:12,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:12,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:12,840 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:12,840 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:12,840 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:12,840 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:12,840 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,840 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,845 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:12,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:12,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,845 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:12,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:12,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:12,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:12,846 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:12,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:12,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:12,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:12,847 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:12,847 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:12,850 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,850 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:12,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5102d6de[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:12,852 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:12,852 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,873 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,873 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:12,873 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:12,873 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:12,873 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:12,874 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:12,874 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:12,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:12,876 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:12,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:12,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:12,876 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:12,876 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:12,877 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,877 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,877 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,878 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321612876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:12,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321612877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,879 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:12,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:12,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:12,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:12,881 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,882 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,881 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:12,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:12,882 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:12,882 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:12,982 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:12,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:12,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:12,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:12,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:12,986 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:12,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:12,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:12,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:12,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,986 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,990 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:12,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:12,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,990 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:12,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:12,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:12,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:12,991 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:12,991 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:12,991 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:12,991 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:12,991 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:12,992 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:12,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:12,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@215e8db5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:12,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:12,994 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,014 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:13,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,014 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:13,015 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:13,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:13,015 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:13,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:13,017 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:13,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:13,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:13,017 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:13,017 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:13,018 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,018 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:13,018 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:13,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:13,019 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:13,020 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:13,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:13,022 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:13,022 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:13,022 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,022 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:13,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:13,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40600 deadline: 1680321613079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 03:59:13,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36251] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40542 deadline: 1680321613080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,122 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:13,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:13,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:13,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:13,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:13,125 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:13,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:13,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:13,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:13,128 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,128 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,140 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:13,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:13,140 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,141 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:13,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:13,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:13,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:13,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:13,141 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:13,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:13,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:13,142 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:13,143 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:13,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:13,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185c9ff5[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:13,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:13,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,144 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:13,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:13,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,144 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:13,144 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:13,145 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:13,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:13,145 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:13,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:13,147 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:13,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:13,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:13,147 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:13,147 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:13,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:13,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:13,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:13,148 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,149 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,150 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,150 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:13,150 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:13,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:13,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:13,150 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:13,150 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:13,150 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,151 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:13,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,151 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:13,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:13,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:13,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:13,151 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,172 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401e8f2aacaec1b43808f972e7af74f8887 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e8f2aacaec1b43808f972e7af74f8887 2023-04-01 03:59:13,188 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/107fcd147b844e8dbff060fabfc36b3c 2023-04-01 03:59:13,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2aaf9420c5b945ae93bb09b36fde7ff4 2023-04-01 03:59:13,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/62e6c769e71a4da08856cc63dac13f1d 2023-04-01 03:59:13,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/107fcd147b844e8dbff060fabfc36b3c as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c 2023-04-01 03:59:13,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c, entries=250, sequenceid=362, filesize=39.2 K 2023-04-01 03:59:13,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2aaf9420c5b945ae93bb09b36fde7ff4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4 2023-04-01 03:59:13,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4, entries=150, sequenceid=362, filesize=11.9 K 2023-04-01 03:59:13,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/62e6c769e71a4da08856cc63dac13f1d as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d 2023-04-01 03:59:13,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d, entries=150, sequenceid=362, filesize=11.9 K 2023-04-01 03:59:13,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 479ms, sequenceid=362, compaction requested=true 2023-04-01 03:59:13,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:13,242 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:13,242 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:13,243 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:13,243 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93006 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:13,243 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A is initiating minor compaction (all files) 2023-04-01 03:59:13,243 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,243 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6d8ed9674b2a4016ae3b52b5bf4de63b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=90.8 K 2023-04-01 03:59:13,243 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B is initiating minor compaction (all files) 2023-04-01 03:59:13,243 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,243 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/7ca5c726462c482d86d643591e2b5192, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=36.5 K 2023-04-01 03:59:13,243 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 6d8ed9674b2a4016ae3b52b5bf4de63b, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:13,244 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 7ca5c726462c482d86d643591e2b5192, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:13,244 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 21dcbffea2da4c2f8424439e9c18511f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680321550533 2023-04-01 03:59:13,244 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2872351d637e449b89bd31f7f0e380e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680321550533 2023-04-01 03:59:13,244 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] compactions.Compactor(207): Compacting 107fcd147b844e8dbff060fabfc36b3c, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680321551628 2023-04-01 03:59:13,244 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 2aaf9420c5b945ae93bb09b36fde7ff4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680321551649 2023-04-01 03:59:13,251 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:13,251 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:13,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:13,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-01 03:59:13,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 03:59:13,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 03:59:13,258 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 03:59:13,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:13,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 03:59:13,258 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 03:59:13,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,259 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,260 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#A#compaction#565 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:13,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,260 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:13,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:13,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:13,260 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 03:59:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 03:59:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:13,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 03:59:13,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 03:59:13,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 03:59:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 03:59:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 03:59:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 03:59:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 03:59:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 03:59:13,262 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,262 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. started... 2023-04-01 03:59:13,262 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 03:59:13,262 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:13,263 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:13,263 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:13,263 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:13,263 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:13,263 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:13,277 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012cfba8ddf1e04c21bd4b119d8dab3073 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cfba8ddf1e04c21bd4b119d8dab3073 2023-04-01 03:59:13,355 DEBUG [Thread-3662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x170f7240 to 127.0.0.1:50457 2023-04-01 03:59:13,355 DEBUG [Thread-3662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,356 DEBUG [Thread-3654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c2e323a to 127.0.0.1:50457 2023-04-01 03:59:13,356 DEBUG [Thread-3660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a953bfa to 127.0.0.1:50457 2023-04-01 03:59:13,357 DEBUG [Thread-3654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,357 DEBUG [Thread-3660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,357 DEBUG [Thread-3656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a9d023 to 127.0.0.1:50457 2023-04-01 03:59:13,357 DEBUG [Thread-3656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,358 DEBUG [Thread-3658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53b986df to 127.0.0.1:50457 2023-04-01 03:59:13,358 DEBUG [Thread-3658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36251] regionserver.HRegion(9158): Flush requested on 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:13,385 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. as already flushing 2023-04-01 03:59:13,385 DEBUG [Thread-3649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4dd8f6f2 to 127.0.0.1:50457 2023-04-01 03:59:13,385 DEBUG [Thread-3649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,385 DEBUG [Thread-3647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29eda94b to 127.0.0.1:50457 2023-04-01 03:59:13,385 DEBUG [Thread-3647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:13,671 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/79c54f8f33df49f78824dc35ee540530 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/79c54f8f33df49f78824dc35ee540530 2023-04-01 03:59:13,677 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/B of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into 79c54f8f33df49f78824dc35ee540530(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:13,677 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:13,677 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, priority=13, startTime=1680321553242; duration=0sec 2023-04-01 03:59:13,678 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 03:59:13,678 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 03:59:13,679 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 03:59:13,679 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1912): 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C is initiating minor compaction (all files) 2023-04-01 03:59:13,679 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C in TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:13,679 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aab836daf4fd4cc6a72942e8c4bc58f9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d] into tmpdir=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp, totalSize=36.5 K 2023-04-01 03:59:13,679 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting aab836daf4fd4cc6a72942e8c4bc58f9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680321550383 2023-04-01 03:59:13,679 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting f93bba7e749042c2b464c6aeed86e3f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680321550533 2023-04-01 03:59:13,680 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] compactions.Compactor(207): Compacting 62e6c769e71a4da08856cc63dac13f1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680321551649 2023-04-01 03:59:13,681 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4 2023-04-01 03:59:13,685 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f6a4fd8a5792a8aca3a5d3a16a6b7a#C#compaction#567 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 03:59:13,691 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2fa8a5d4b6df4a11800b12a7ac7bab62 2023-04-01 03:59:14,105 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a 2023-04-01 03:59:14,105 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/e91347bce4c0474986140005c70e31e6 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e91347bce4c0474986140005c70e31e6 2023-04-01 03:59:14,108 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4 2023-04-01 03:59:14,109 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/C of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into e91347bce4c0474986140005c70e31e6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 03:59:14,109 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:14,109 INFO [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, priority=13, startTime=1680321553242; duration=0sec 2023-04-01 03:59:14,109 DEBUG [RS:0;jenkins-hbase20:36251-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:14,110 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4, entries=150, sequenceid=375, filesize=25.4 K 2023-04-01 03:59:14,110 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/2fa8a5d4b6df4a11800b12a7ac7bab62 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2fa8a5d4b6df4a11800b12a7ac7bab62 2023-04-01 03:59:14,113 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2fa8a5d4b6df4a11800b12a7ac7bab62, entries=150, sequenceid=375, filesize=11.9 K 2023-04-01 03:59:14,113 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a 2023-04-01 03:59:14,116 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a, entries=150, sequenceid=375, filesize=11.9 K 2023-04-01 03:59:14,116 INFO [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=13.42 KB/13740 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 854ms, sequenceid=375, compaction requested=false 2023-04-01 03:59:14,116 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:14,117 DEBUG [rs(jenkins-hbase20.apache.org,36251,1680321375816)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:14,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 03:59:14,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 03:59:14,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 03:59:14,117 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 03:59:14,118 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:14,118 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:14,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36251,1680321375816' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 03:59:14,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2559fea7[Count = 0] remaining members to acquire global barrier 2023-04-01 03:59:14,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 03:59:14,119 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,120 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 03:59:14,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 03:59:14,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,120 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 03:59:14,120 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36251,1680321375816' in zk 2023-04-01 03:59:14,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:14,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:14,122 DEBUG [member: 'jenkins-hbase20.apache.org,36251,1680321375816' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36251,1680321375816': 2023-04-01 03:59:14,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36251,1680321375816' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 03:59:14,123 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 03:59:14,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 03:59:14,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 03:59:14,123 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 03:59:14,123 INFO [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 03:59:14,125 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,125 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 03:59:14,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,125 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 03:59:14,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:14,125 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,126 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 03:59:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 03:59:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 03:59:14,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 03:59:14,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 03:59:14,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 03:59:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 03:59:14,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 03:59:14,135 DEBUG [Thread-3653] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 03:59:14,135 DEBUG [Thread-3653] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 03:59:14,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 03:59:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 03:59:14,135 DEBUG [(jenkins-hbase20.apache.org,34393,1680321374904)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:14,135 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 03:59:14,136 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 03:59:14,235 DEBUG [Thread-3653] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 03:59:14,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 03:59:14,495 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/d31c2b16bfa84c158fa2b9a81b50dfaa as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/d31c2b16bfa84c158fa2b9a81b50dfaa 2023-04-01 03:59:14,503 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f6a4fd8a5792a8aca3a5d3a16a6b7a/A of 08f6a4fd8a5792a8aca3a5d3a16a6b7a into d31c2b16bfa84c158fa2b9a81b50dfaa(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 03:59:14,504 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:14,504 INFO [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a., storeName=08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, priority=13, startTime=1680321553242; duration=1sec 2023-04-01 03:59:14,504 DEBUG [RS:0;jenkins-hbase20:36251-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 03:59:14,671 DEBUG [Thread-3651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07e01249 to 127.0.0.1:50457 2023-04-01 03:59:14,671 DEBUG [Thread-3651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:17,375 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-01 03:59:20,822 DEBUG [Thread-3643] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2145bb3e to 127.0.0.1:50457 2023-04-01 03:59:20,822 DEBUG [Thread-3643] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:20,853 DEBUG [Thread-3645] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c268ca to 127.0.0.1:50457 2023-04-01 03:59:20,854 DEBUG [Thread-3645] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5685 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5729 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5705 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5733 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(398): read 5734 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 03:59:20,854 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:59:20,854 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2da434f4 to 127.0.0.1:50457 2023-04-01 03:59:20,854 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:20,855 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 03:59:20,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-01 03:59:20,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:20,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 03:59:20,857 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321560857"}]},"ts":"1680321560857"} 2023-04-01 03:59:20,858 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 03:59:20,878 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 03:59:20,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, UNASSIGN}] 2023-04-01 03:59:20,880 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, UNASSIGN 2023-04-01 03:59:20,880 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:20,880 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321560880"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680321560880"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680321560880"}]},"ts":"1680321560880"} 2023-04-01 03:59:20,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816}] 2023-04-01 03:59:20,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 03:59:21,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08f6a4fd8a5792a8aca3a5d3a16a6b7a, disabling compactions & flushes 2023-04-01 03:59:21,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. after waiting 0 ms 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:21,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 08f6a4fd8a5792a8aca3a5d3a16a6b7a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=A 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=B 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:21,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f6a4fd8a5792a8aca3a5d3a16a6b7a, store=C 2023-04-01 03:59:21,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 03:59:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 03:59:21,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401569de578b5b74e198f48b7c5bfa98090 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401569de578b5b74e198f48b7c5bfa98090 2023-04-01 03:59:21,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=386, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7a73b0ea31794741b4c6f5477120a8dc 2023-04-01 03:59:21,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 03:59:21,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/dc1af442a9b34b638ff18bb2dfd320fd 2023-04-01 03:59:21,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/fca3c714442644c3ae4ade56841f3976 2023-04-01 03:59:21,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/A/7a73b0ea31794741b4c6f5477120a8dc as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7a73b0ea31794741b4c6f5477120a8dc 2023-04-01 03:59:21,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7a73b0ea31794741b4c6f5477120a8dc, entries=150, sequenceid=386, filesize=25.4 K 2023-04-01 03:59:21,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/B/dc1af442a9b34b638ff18bb2dfd320fd as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/dc1af442a9b34b638ff18bb2dfd320fd 2023-04-01 03:59:21,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/dc1af442a9b34b638ff18bb2dfd320fd, entries=150, sequenceid=386, filesize=11.9 K 2023-04-01 03:59:21,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/.tmp/C/fca3c714442644c3ae4ade56841f3976 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fca3c714442644c3ae4ade56841f3976 2023-04-01 03:59:21,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fca3c714442644c3ae4ade56841f3976, entries=150, sequenceid=386, filesize=11.9 K 2023-04-01 03:59:21,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 08f6a4fd8a5792a8aca3a5d3a16a6b7a in 472ms, sequenceid=386, compaction requested=true 2023-04-01 03:59:21,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/b18e633b9de24abe95c264517b0531bd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7373cb945a93497fb740de3fb6a45665, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0fab7bd5d6ba448398b23046b3b20eba, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/41cc56d4905f433ca5821aedbf9d05a9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/e346f9cd864645179620f499800af6e6, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6d8ed9674b2a4016ae3b52b5bf4de63b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c] to archive 2023-04-01 03:59:21,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:59:21,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7e0e1d359bb94c879a8157d89db8a311 2023-04-01 03:59:21,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c6865f90269b4aa69dbf7771e0f8670d 2023-04-01 03:59:21,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/b18e633b9de24abe95c264517b0531bd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/b18e633b9de24abe95c264517b0531bd 2023-04-01 03:59:21,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/567bb6066f44404cb472e826c37f1f6f 2023-04-01 03:59:21,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c1fdf73b8bf847c8909e55376d8c9778 2023-04-01 03:59:21,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9ae9021e9a074042b5cdf0ae65d4b0a8 2023-04-01 03:59:21,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7373cb945a93497fb740de3fb6a45665 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7373cb945a93497fb740de3fb6a45665 2023-04-01 03:59:21,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9fc6722f0b72427c8d04a7690ea0e03f 2023-04-01 03:59:21,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/c07d3463a7934669ac3fa39562f94aa0 2023-04-01 03:59:21,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/2b981ec2c5394452a5c63b3030271d63 2023-04-01 03:59:21,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0fab7bd5d6ba448398b23046b3b20eba to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0fab7bd5d6ba448398b23046b3b20eba 2023-04-01 03:59:21,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5e0ec02bb9ce4d498f3503cf1431ae55 2023-04-01 03:59:21,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9b8bc49083fb4463be86d099d9366c00 2023-04-01 03:59:21,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/5cd345d9c2df42b7a8a2a5c45419b92b 2023-04-01 03:59:21,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/41cc56d4905f433ca5821aedbf9d05a9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/41cc56d4905f433ca5821aedbf9d05a9 2023-04-01 03:59:21,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/410fb013b5c8466f88656d9b859b650e 2023-04-01 03:59:21,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7cd6570db3b54f6a883df066ada44ee9 2023-04-01 03:59:21,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/e346f9cd864645179620f499800af6e6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/e346f9cd864645179620f499800af6e6 2023-04-01 03:59:21,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6182ba014eb74f9baafd9905f0b5d55c 2023-04-01 03:59:21,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/98a297878a564de19a1fed96027f5514 2023-04-01 03:59:21,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/89d4b2a9a5fe4f4487c940c45bf00599 2023-04-01 03:59:21,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/0578f1ca749b4a9eac75c6279351fd36 2023-04-01 03:59:21,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/431502e6401544529288c67e5c22acf0 2023-04-01 03:59:21,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6d8ed9674b2a4016ae3b52b5bf4de63b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/6d8ed9674b2a4016ae3b52b5bf4de63b 2023-04-01 03:59:21,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/21dcbffea2da4c2f8424439e9c18511f 2023-04-01 03:59:21,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/107fcd147b844e8dbff060fabfc36b3c 2023-04-01 03:59:21,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e8f1f56b9bf345aabdd778522c45ebf8, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/9b2cb7062cfc4ac9b0882a293e5dd699, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d4e6e5ce17664d1db299ac2212e99169, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0e060f47ba0b4e2ead1ad7581cd2c0d1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb02ee12c36543e690ac89657eaef1a2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/7ca5c726462c482d86d643591e2b5192, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4] to archive 2023-04-01 03:59:21,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:59:21,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0f69f4e2ac6548c6b06570732135de86 2023-04-01 03:59:21,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/bc917ac4731e4b18a5d19a6cb0071916 2023-04-01 03:59:21,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e8f1f56b9bf345aabdd778522c45ebf8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e8f1f56b9bf345aabdd778522c45ebf8 2023-04-01 03:59:21,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e04d83af8a70445ba2832eeb30a2353a 2023-04-01 03:59:21,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/16227592f8fc4adca4ca53a6cbd9f592 2023-04-01 03:59:21,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/6124690a6f064e628e8889c6b0e21c5a 2023-04-01 03:59:21,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/9b2cb7062cfc4ac9b0882a293e5dd699 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/9b2cb7062cfc4ac9b0882a293e5dd699 2023-04-01 03:59:21,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/e28412a411404725b294ee57a86d5641 2023-04-01 03:59:21,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/3700e9a3c2954e8db1bc0a0fbfc51cf0 2023-04-01 03:59:21,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/a7640177b3aa469192e13745c55adef1 2023-04-01 03:59:21,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d4e6e5ce17664d1db299ac2212e99169 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d4e6e5ce17664d1db299ac2212e99169 2023-04-01 03:59:21,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/f679c79c3e16477ba5341526a3775a7c 2023-04-01 03:59:21,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/703de9660d68471481fbdd4b770cd590 2023-04-01 03:59:21,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0e060f47ba0b4e2ead1ad7581cd2c0d1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/0e060f47ba0b4e2ead1ad7581cd2c0d1 2023-04-01 03:59:21,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/228f659a5c1749f490054e5fb557f9f4 2023-04-01 03:59:21,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb72ba60b0ed4376bb180f87f1e6016b 2023-04-01 03:59:21,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb02ee12c36543e690ac89657eaef1a2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/eb02ee12c36543e690ac89657eaef1a2 2023-04-01 03:59:21,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/747aed9faaab4ff6ae0ae47e2fed17fd 2023-04-01 03:59:21,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/cd44a504d9414517b1d9cdc580f469e9 2023-04-01 03:59:21,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/135ea3240427437ab6d3177039fa5dd2 2023-04-01 03:59:21,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/c777d448f133465194eedf6a449af919 2023-04-01 03:59:21,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/d89cc3019277418eb143f3a3caecea23 2023-04-01 03:59:21,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/7ca5c726462c482d86d643591e2b5192 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/7ca5c726462c482d86d643591e2b5192 2023-04-01 03:59:21,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/29a27216375f47b4aa43f1807966cbee 2023-04-01 03:59:21,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2872351d637e449b89bd31f7f0e380e9 2023-04-01 03:59:21,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2aaf9420c5b945ae93bb09b36fde7ff4 2023-04-01 03:59:21,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/75b0ea63f1b547429e8efa9609eaf370, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/ddc3d0797a37455facb8b49546b2b389, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e4f34803826b478b88cd7ad32e5c8203, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/9fe99134d6884ca68c2632ad709a6fc3, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/a60f759c73634b2a8169870e64dc6eb1, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aab836daf4fd4cc6a72942e8c4bc58f9, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d] to archive 2023-04-01 03:59:21,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 03:59:21,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fdd3179c4a5a4b4d858b06753b55be36 2023-04-01 03:59:21,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aa083c6cd9c6489ea0d443b2791d77ae 2023-04-01 03:59:21,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/75b0ea63f1b547429e8efa9609eaf370 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/75b0ea63f1b547429e8efa9609eaf370 2023-04-01 03:59:21,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/66486068c07f4d03bfe67a9a143f115f 2023-04-01 03:59:21,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e951a4a97657488eb8aafb6c6108d169 2023-04-01 03:59:21,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4e93d7a06c144b57a3acc97584803a1c 2023-04-01 03:59:21,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/ddc3d0797a37455facb8b49546b2b389 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/ddc3d0797a37455facb8b49546b2b389 2023-04-01 03:59:21,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/789d31fb592043429de7840cb30ced25 2023-04-01 03:59:21,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/7a99476a5b40484fa3cd9fc4c85d0b91 2023-04-01 03:59:21,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/094de775ff7d4d67804a7cc4951c279a 2023-04-01 03:59:21,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e4f34803826b478b88cd7ad32e5c8203 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e4f34803826b478b88cd7ad32e5c8203 2023-04-01 03:59:21,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1387ddad4d564af88e1fff37d6dd1b23 2023-04-01 03:59:21,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/b2559681e9de4d87860d1ef41a44faa4 2023-04-01 03:59:21,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/9fe99134d6884ca68c2632ad709a6fc3 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/9fe99134d6884ca68c2632ad709a6fc3 2023-04-01 03:59:21,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/93598f02d8ad44abad2b828cb038d8e9 2023-04-01 03:59:21,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/03ad7a6bd32142eb834daf69a18f5684 2023-04-01 03:59:21,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/a60f759c73634b2a8169870e64dc6eb1 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/a60f759c73634b2a8169870e64dc6eb1 2023-04-01 03:59:21,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4fa05c23fe574680bad94c815a03e959 2023-04-01 03:59:21,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/653977957503447b85fbbdab3cfbbb87 2023-04-01 03:59:21,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/2ec031c8c6524ee698e43a8c80e6af69 2023-04-01 03:59:21,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/dde6ff05ce734554bb33fd3bc0a9eb0d 2023-04-01 03:59:21,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/bec02507afd9456d94fd5836aca3b1fc 2023-04-01 03:59:21,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aab836daf4fd4cc6a72942e8c4bc58f9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/aab836daf4fd4cc6a72942e8c4bc58f9 2023-04-01 03:59:21,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/1f35eec1278e42e7a20db08cfd027ff5 2023-04-01 03:59:21,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/f93bba7e749042c2b464c6aeed86e3f2 2023-04-01 03:59:21,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/62e6c769e71a4da08856cc63dac13f1d 2023-04-01 03:59:21,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits/389.seqid, newMaxSeqId=389, maxSeqId=4 2023-04-01 03:59:21,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a. 2023-04-01 03:59:21,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08f6a4fd8a5792a8aca3a5d3a16a6b7a: 2023-04-01 03:59:21,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:21,606 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=08f6a4fd8a5792a8aca3a5d3a16a6b7a, regionState=CLOSED 2023-04-01 03:59:21,606 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680321561606"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680321561606"}]},"ts":"1680321561606"} 2023-04-01 03:59:21,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-01 03:59:21,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 08f6a4fd8a5792a8aca3a5d3a16a6b7a, server=jenkins-hbase20.apache.org,36251,1680321375816 in 725 msec 2023-04-01 03:59:21,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-01 03:59:21,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f6a4fd8a5792a8aca3a5d3a16a6b7a, UNASSIGN in 728 msec 2023-04-01 03:59:21,608 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680321561608"}]},"ts":"1680321561608"} 2023-04-01 03:59:21,609 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 03:59:21,617 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 03:59:21,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 762 msec 2023-04-01 03:59:21,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 03:59:21,962 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-01 03:59:21,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-01 03:59:21,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:21,967 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:21,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-01 03:59:21,968 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:21,973 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:21,975 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C, FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits] 2023-04-01 03:59:21,977 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7a73b0ea31794741b4c6f5477120a8dc to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/7a73b0ea31794741b4c6f5477120a8dc 2023-04-01 03:59:21,979 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/9baa65bc7bdd49cc9dd2f5d7ae3c3fa4 2023-04-01 03:59:21,980 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/d31c2b16bfa84c158fa2b9a81b50dfaa to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/A/d31c2b16bfa84c158fa2b9a81b50dfaa 2023-04-01 03:59:21,982 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2fa8a5d4b6df4a11800b12a7ac7bab62 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/2fa8a5d4b6df4a11800b12a7ac7bab62 2023-04-01 03:59:21,983 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/79c54f8f33df49f78824dc35ee540530 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/79c54f8f33df49f78824dc35ee540530 2023-04-01 03:59:21,984 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/dc1af442a9b34b638ff18bb2dfd320fd to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/B/dc1af442a9b34b638ff18bb2dfd320fd 2023-04-01 03:59:21,987 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/4dbf2db2c7b04f72ba5677f0e7dfcd6a 2023-04-01 03:59:21,989 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e91347bce4c0474986140005c70e31e6 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/e91347bce4c0474986140005c70e31e6 2023-04-01 03:59:21,990 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fca3c714442644c3ae4ade56841f3976 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/C/fca3c714442644c3ae4ade56841f3976 2023-04-01 03:59:21,993 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits/389.seqid to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a/recovered.edits/389.seqid 2023-04-01 03:59:21,993 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/.tmp/data/default/TestAcidGuarantees/08f6a4fd8a5792a8aca3a5d3a16a6b7a 2023-04-01 03:59:21,993 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 03:59:21,994 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:59:21,994 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 03:59:21,997 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401055647ff87474ec9a09bdf9f5d576b01 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401055647ff87474ec9a09bdf9f5d576b01 2023-04-01 03:59:21,998 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011332caedbd6548778d81ff4a345147b8 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011332caedbd6548778d81ff4a345147b8 2023-04-01 03:59:22,000 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cfba8ddf1e04c21bd4b119d8dab3073 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012cfba8ddf1e04c21bd4b119d8dab3073 2023-04-01 03:59:22,001 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040135878260ca5c4bea833aaa510cb6379c to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040135878260ca5c4bea833aaa510cb6379c 2023-04-01 03:59:22,002 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040140a57b7123524826885b9fa0c143ef77 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040140a57b7123524826885b9fa0c143ef77 2023-04-01 03:59:22,003 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401569de578b5b74e198f48b7c5bfa98090 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401569de578b5b74e198f48b7c5bfa98090 2023-04-01 03:59:22,004 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015a5d7bf9726b4bffa686cd9a06684a43 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015a5d7bf9726b4bffa686cd9a06684a43 2023-04-01 03:59:22,005 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040160a1fdb40b1a486896deefbcb652b889 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040160a1fdb40b1a486896deefbcb652b889 2023-04-01 03:59:22,006 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401756b71dd6dc24857bab7b13d9ae30ee9 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401756b71dd6dc24857bab7b13d9ae30ee9 2023-04-01 03:59:22,008 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017b453f9ab5ba45bb86ed571365300c42 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017b453f9ab5ba45bb86ed571365300c42 2023-04-01 03:59:22,009 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401840fa3c7f50f45729daf1c16bc052837 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401840fa3c7f50f45729daf1c16bc052837 2023-04-01 03:59:22,010 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018f01a1d3dd424739acc8237cd0b9871d to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018f01a1d3dd424739acc8237cd0b9871d 2023-04-01 03:59:22,011 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019c2c502385174913b46f4f102f77fa67 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019c2c502385174913b46f4f102f77fa67 2023-04-01 03:59:22,012 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a8350a1177614b8d8905660fec67e00b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a8350a1177614b8d8905660fec67e00b 2023-04-01 03:59:22,013 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aba5e607cc2f44ac9923255e71515758 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401aba5e607cc2f44ac9923255e71515758 2023-04-01 03:59:22,014 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401adfd589bfdc947fcb30a717d1c709050 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401adfd589bfdc947fcb30a717d1c709050 2023-04-01 03:59:22,016 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401caa211a2c9194d51ac6353e9679ee3bf to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401caa211a2c9194d51ac6353e9679ee3bf 2023-04-01 03:59:22,017 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401dd420a2e771744db9bea5f7a276d287b to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401dd420a2e771744db9bea5f7a276d287b 2023-04-01 03:59:22,019 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e4d0d182e6944a4e832d853aea8ed486 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e4d0d182e6944a4e832d853aea8ed486 2023-04-01 03:59:22,020 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e8f2aacaec1b43808f972e7af74f8887 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e8f2aacaec1b43808f972e7af74f8887 2023-04-01 03:59:22,022 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f4666d44a7af4c7d9242b2f6e591f315 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f4666d44a7af4c7d9242b2f6e591f315 2023-04-01 03:59:22,023 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fb17a4b80456423a866f66a3ec98be93 to hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fb17a4b80456423a866f66a3ec98be93 2023-04-01 03:59:22,023 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 03:59:22,026 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:22,027 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 03:59:22,029 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 03:59:22,030 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:22,030 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 03:59:22,030 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680321562030"}]},"ts":"9223372036854775807"} 2023-04-01 03:59:22,031 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 03:59:22,031 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 08f6a4fd8a5792a8aca3a5d3a16a6b7a, NAME => 'TestAcidGuarantees,,1680321531252.08f6a4fd8a5792a8aca3a5d3a16a6b7a.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 03:59:22,032 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 03:59:22,032 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680321562032"}]},"ts":"9223372036854775807"} 2023-04-01 03:59:22,033 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 03:59:22,035 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 03:59:22,037 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 72 msec 2023-04-01 03:59:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34393] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-01 03:59:22,069 INFO [Listener at localhost.localdomain/39075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-01 03:59:22,085 INFO [Listener at localhost.localdomain/39075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 227), OpenFileDescriptor=532 (was 547), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=482 (was 625), ProcessCount=167 (was 169), AvailableMemoryMB=8678 (was 9044) 2023-04-01 03:59:22,086 INFO [Listener at localhost.localdomain/39075] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-01 03:59:22,086 INFO [Listener at localhost.localdomain/39075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 03:59:22,086 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37e2243f to 127.0.0.1:50457 2023-04-01 03:59:22,086 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:22,086 DEBUG [Listener at localhost.localdomain/39075] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-01 03:59:22,086 DEBUG [Listener at localhost.localdomain/39075] util.JVMClusterUtil(257): Found active master hash=185310104, stopped=false 2023-04-01 03:59:22,086 INFO [Listener at localhost.localdomain/39075] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:59:22,088 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-01 03:59:22,088 INFO [Listener at localhost.localdomain/39075] procedure2.ProcedureExecutor(629): Stopping 2023-04-01 03:59:22,088 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-01 03:59:22,088 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:59:22,088 DEBUG [Listener at localhost.localdomain/39075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f9fda4d to 127.0.0.1:50457 2023-04-01 03:59:22,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 03:59:22,088 DEBUG [Listener at localhost.localdomain/39075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:22,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 03:59:22,089 INFO [Listener at localhost.localdomain/39075] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,36251,1680321375816' ***** 2023-04-01 03:59:22,089 INFO [Listener at localhost.localdomain/39075] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-01 03:59:22,089 INFO [RS:0;jenkins-hbase20:36251] regionserver.HeapMemoryManager(220): Stopping 2023-04-01 03:59:22,089 INFO [RS:0;jenkins-hbase20:36251] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-01 03:59:22,089 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-01 03:59:22,089 INFO [RS:0;jenkins-hbase20:36251] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-01 03:59:22,089 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(3304): Received CLOSE for 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:59:22,090 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:22,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 9831deae690ed99cd6a015ff582234aa, disabling compactions & flushes 2023-04-01 03:59:22,090 DEBUG [RS:0;jenkins-hbase20:36251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76311fa0 to 127.0.0.1:50457 2023-04-01 03:59:22,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:59:22,091 DEBUG [RS:0;jenkins-hbase20:36251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:22,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:59:22,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. after waiting 0 ms 2023-04-01 03:59:22,091 INFO [RS:0;jenkins-hbase20:36251] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-01 03:59:22,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:59:22,091 INFO [RS:0;jenkins-hbase20:36251] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-01 03:59:22,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 9831deae690ed99cd6a015ff582234aa 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-01 03:59:22,091 INFO [RS:0;jenkins-hbase20:36251] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-01 03:59:22,091 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-01 03:59:22,091 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-01 03:59:22,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-01 03:59:22,092 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1479): Online Regions={9831deae690ed99cd6a015ff582234aa=hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa., 1588230740=hbase:meta,,1.1588230740} 2023-04-01 03:59:22,092 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-01 03:59:22,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-01 03:59:22,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-01 03:59:22,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-01 03:59:22,092 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-01 03:59:22,094 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1505): Waiting on 1588230740, 9831deae690ed99cd6a015ff582234aa 2023-04-01 03:59:22,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/.tmp/info/a94b69c09aa447be957d94a3dfa361a8 2023-04-01 03:59:22,099 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/info/9eef13155e90435fa95e072b80c204f5 2023-04-01 03:59:22,108 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9eef13155e90435fa95e072b80c204f5 2023-04-01 03:59:22,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/.tmp/info/a94b69c09aa447be957d94a3dfa361a8 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/info/a94b69c09aa447be957d94a3dfa361a8 2023-04-01 03:59:22,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/info/a94b69c09aa447be957d94a3dfa361a8, entries=2, sequenceid=6, filesize=4.8 K 2023-04-01 03:59:22,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9831deae690ed99cd6a015ff582234aa in 22ms, sequenceid=6, compaction requested=false 2023-04-01 03:59:22,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/namespace/9831deae690ed99cd6a015ff582234aa/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-01 03:59:22,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:59:22,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 9831deae690ed99cd6a015ff582234aa: 2023-04-01 03:59:22,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680321378260.9831deae690ed99cd6a015ff582234aa. 2023-04-01 03:59:22,134 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-01 03:59:22,294 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 03:59:22,348 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-01 03:59:22,349 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-01 03:59:22,495 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 03:59:22,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/rep_barrier/95ffcf6ac0cc414e82d83e5bac0e0906 2023-04-01 03:59:22,523 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95ffcf6ac0cc414e82d83e5bac0e0906 2023-04-01 03:59:22,695 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 03:59:22,895 DEBUG [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 03:59:22,933 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/table/967742ae208c434794d7a418f94c68f4 2023-04-01 03:59:22,941 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 967742ae208c434794d7a418f94c68f4 2023-04-01 03:59:22,942 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/info/9eef13155e90435fa95e072b80c204f5 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/info/9eef13155e90435fa95e072b80c204f5 2023-04-01 03:59:22,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9eef13155e90435fa95e072b80c204f5 2023-04-01 03:59:22,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/info/9eef13155e90435fa95e072b80c204f5, entries=22, sequenceid=93, filesize=7.1 K 2023-04-01 03:59:22,952 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/rep_barrier/95ffcf6ac0cc414e82d83e5bac0e0906 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/rep_barrier/95ffcf6ac0cc414e82d83e5bac0e0906 2023-04-01 03:59:22,958 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95ffcf6ac0cc414e82d83e5bac0e0906 2023-04-01 03:59:22,958 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/rep_barrier/95ffcf6ac0cc414e82d83e5bac0e0906, entries=6, sequenceid=93, filesize=5.5 K 2023-04-01 03:59:22,959 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/.tmp/table/967742ae208c434794d7a418f94c68f4 as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/table/967742ae208c434794d7a418f94c68f4 2023-04-01 03:59:22,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 967742ae208c434794d7a418f94c68f4 2023-04-01 03:59:22,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/table/967742ae208c434794d7a418f94c68f4, entries=9, sequenceid=93, filesize=5.5 K 2023-04-01 03:59:22,965 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 873ms, sequenceid=93, compaction requested=false 2023-04-01 03:59:22,978 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-01 03:59:22,979 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-01 03:59:22,980 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-01 03:59:22,980 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-01 03:59:22,980 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-01 03:59:23,095 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,36251,1680321375816; all regions closed. 2023-04-01 03:59:23,103 DEBUG [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/oldWALs 2023-04-01 03:59:23,103 INFO [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36251%2C1680321375816.meta:.meta(num 1680321378028) 2023-04-01 03:59:23,109 DEBUG [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/oldWALs 2023-04-01 03:59:23,109 INFO [RS:0;jenkins-hbase20:36251] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36251%2C1680321375816:(num 1680321377441) 2023-04-01 03:59:23,109 DEBUG [RS:0;jenkins-hbase20:36251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:23,109 INFO [RS:0;jenkins-hbase20:36251] regionserver.LeaseManager(133): Closed leases 2023-04-01 03:59:23,109 INFO [RS:0;jenkins-hbase20:36251] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-01 03:59:23,110 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-01 03:59:23,111 INFO [RS:0;jenkins-hbase20:36251] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36251 2023-04-01 03:59:23,116 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36251,1680321375816 2023-04-01 03:59:23,117 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38e52544 rejected from java.util.concurrent.ThreadPoolExecutor@3cdab680[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2241] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 03:59:23,117 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5935a0e9 rejected from java.util.concurrent.ThreadPoolExecutor@3cdab680[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2241] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,117 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 03:59:23,118 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,36251,1680321375816] 2023-04-01 03:59:23,119 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,36251,1680321375816; numProcessing=1 2023-04-01 03:59:23,120 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,36251,1680321375816 already deleted, retry=false 2023-04-01 03:59:23,120 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,36251,1680321375816 expired; onlineServers=0 2023-04-01 03:59:23,121 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34393,1680321374904' ***** 2023-04-01 03:59:23,121 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-01 03:59:23,121 DEBUG [M:0;jenkins-hbase20:34393] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a3691a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-01 03:59:23,121 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34393,1680321374904 2023-04-01 03:59:23,121 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34393,1680321374904; all regions closed. 2023-04-01 03:59:23,121 DEBUG [M:0;jenkins-hbase20:34393] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 03:59:23,121 DEBUG [M:0;jenkins-hbase20:34393] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-01 03:59:23,121 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-01 03:59:23,121 DEBUG [M:0;jenkins-hbase20:34393] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-01 03:59:23,121 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680321377143] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680321377143,5,FailOnTimeoutGroup] 2023-04-01 03:59:23,121 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680321377142] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680321377142,5,FailOnTimeoutGroup] 2023-04-01 03:59:23,121 INFO [M:0;jenkins-hbase20:34393] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-01 03:59:23,121 INFO [M:0;jenkins-hbase20:34393] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-01 03:59:23,122 INFO [M:0;jenkins-hbase20:34393] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-01 03:59:23,122 DEBUG [M:0;jenkins-hbase20:34393] master.HMaster(1512): Stopping service threads 2023-04-01 03:59:23,122 INFO [M:0;jenkins-hbase20:34393] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-01 03:59:23,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-01 03:59:23,122 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 03:59:23,123 ERROR [M:0;jenkins-hbase20:34393] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-01 03:59:23,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 03:59:23,123 INFO [M:0;jenkins-hbase20:34393] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-01 03:59:23,123 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-01 03:59:23,124 DEBUG [M:0;jenkins-hbase20:34393] zookeeper.ZKUtil(398): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-01 03:59:23,124 WARN [M:0;jenkins-hbase20:34393] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-01 03:59:23,124 INFO [M:0;jenkins-hbase20:34393] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-01 03:59:23,124 INFO [M:0;jenkins-hbase20:34393] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-01 03:59:23,125 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-01 03:59:23,125 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:59:23,125 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:59:23,125 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-01 03:59:23,125 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:59:23,125 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.32 KB heapSize=393.95 KB 2023-04-01 03:59:23,219 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 03:59:23,219 INFO [RS:0;jenkins-hbase20:36251] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,36251,1680321375816; zookeeper connection closed. 2023-04-01 03:59:23,219 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@216ca0b5 rejected from java.util.concurrent.ThreadPoolExecutor@3cdab680[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2241] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,220 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): regionserver:36251-0x1006aab7ff40001, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 03:59:23,220 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6db1090f rejected from java.util.concurrent.ThreadPoolExecutor@3cdab680[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2241] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,220 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1f4a7d1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1f4a7d1 2023-04-01 03:59:23,220 INFO [Listener at localhost.localdomain/39075] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-01 03:59:23,537 INFO [M:0;jenkins-hbase20:34393] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.32 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f3a82303bf654d17bd0d6d7bcabc417f 2023-04-01 03:59:23,546 INFO [M:0;jenkins-hbase20:34393] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f3a82303bf654d17bd0d6d7bcabc417f 2023-04-01 03:59:23,547 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f3a82303bf654d17bd0d6d7bcabc417f as hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f3a82303bf654d17bd0d6d7bcabc417f 2023-04-01 03:59:23,552 INFO [M:0;jenkins-hbase20:34393] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f3a82303bf654d17bd0d6d7bcabc417f 2023-04-01 03:59:23,552 INFO [M:0;jenkins-hbase20:34393] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32871/user/jenkins/test-data/d668e841-09f4-bdc2-c2f5-c26d9da7763a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f3a82303bf654d17bd0d6d7bcabc417f, entries=65, sequenceid=578, filesize=24.7 K 2023-04-01 03:59:23,553 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegion(2948): Finished flush of dataSize ~345.32 KB/353611, heapSize ~393.94 KB/403392, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 428ms, sequenceid=578, compaction requested=false 2023-04-01 03:59:23,555 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 03:59:23,555 DEBUG [M:0;jenkins-hbase20:34393] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 03:59:23,558 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-01 03:59:23,558 INFO [M:0;jenkins-hbase20:34393] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-01 03:59:23,559 INFO [M:0;jenkins-hbase20:34393] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34393 2023-04-01 03:59:23,561 DEBUG [M:0;jenkins-hbase20:34393] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,34393,1680321374904 already deleted, retry=false 2023-04-01 03:59:23,662 INFO [M:0;jenkins-hbase20:34393] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34393,1680321374904; zookeeper connection closed. 2023-04-01 03:59:23,662 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 03:59:23,662 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@359b47e7 rejected from java.util.concurrent.ThreadPoolExecutor@4f8a3d4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3757] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,663 DEBUG [Listener at localhost.localdomain/39075-EventThread] zookeeper.ZKWatcher(600): master:34393-0x1006aab7ff40000, quorum=127.0.0.1:50457, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 03:59:23,663 ERROR [Listener at localhost.localdomain/39075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@253b07cf rejected from java.util.concurrent.ThreadPoolExecutor@4f8a3d4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3757] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 03:59:23,664 WARN [Listener at localhost.localdomain/39075] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-01 03:59:23,673 INFO [Listener at localhost.localdomain/39075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-01 03:59:23,781 WARN [BP-740163357-148.251.75.209-1680321372222 heartbeating to localhost.localdomain/127.0.0.1:32871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-01 03:59:23,781 WARN [BP-740163357-148.251.75.209-1680321372222 heartbeating to localhost.localdomain/127.0.0.1:32871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-740163357-148.251.75.209-1680321372222 (Datanode Uuid 1765434b-9800-4ee8-9fa5-09d8b0e4da2f) service to localhost.localdomain/127.0.0.1:32871 2023-04-01 03:59:23,785 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648/dfs/data/data1/current/BP-740163357-148.251.75.209-1680321372222] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-01 03:59:23,786 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d85a2548-548c-79b4-0f41-05b88d54064e/cluster_6acc2dc7-79e4-4f2e-d4b8-9431e438f648/dfs/data/data2/current/BP-740163357-148.251.75.209-1680321372222] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-01 03:59:23,817 INFO [Listener at localhost.localdomain/39075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-01 03:59:23,972 INFO [Listener at localhost.localdomain/39075] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-01 03:59:24,133 INFO [Listener at localhost.localdomain/39075] hbase.HBaseTestingUtility(1293): Minicluster is down